OpenRailAssociation / osrd

An open source web application for railway infrastructure design, capacity analysis, timetabling and simulation
https://osrd.fr
439 stars 39 forks source link

front: log and make E2E tests fail on uncaught exceptions #8883

Open emersion opened 4 days ago

emersion commented 4 days ago

Description and goal

Right now when an E2E test fails we don't have lots of details. When it trips on an exception we instead get a timeout and no details.

See:

Acceptance criteria

.

Maymanaf commented 4 days ago

It's true that we currently lack detailed logs when an E2E test fails, but for now, we can review the screen recordings and trace files available in the CI artifacts.

Planned Solution: We will implement a new reporter (Allure Report) and integrate custom logs for each test.

:warning: This task is not prioritized at the moment

emersion commented 4 days ago

I don't think changing the reporter would help. Even if it reports uncaught exceptions (does it?), tests triggering uncaught exceptions would still succeed (but we want these to fail).

Maymanaf commented 4 days ago

As you mentioned in the ticket, we can use pageerror to handle uncaught exceptions. However, these need to be customized and clearly defined, which is why we're considering a different reporting approach. That said, I can add a non-optimized solution to fail the tests. Did you notice many uncaught exceptions not failing the tests?