Correlate unittest results with video timestamps in openQA

marmarek commented 3 years ago

The problem you're addressing (if any)

Getting more (visual) details what has failed requires watching the whole test video recording (well, a kind of binary search in practice, but still inconvenient).

The solution you'd like

Add links to specific timestamps next to test results. For openQA native tests, every screenshot has a link to specific video timestamp (see https://openqa.qubes-os.org/tests/22143#step/system_tests/4 for example). The idea is to add similar one for python unittests (which we run via nose2 runner).

This require at least:

adding timestamp information to the junit xml file (output of nose2 runner) - currently there is only test duration, but not absolute start time of individual tests
convince openQA to link it (may require extending openQA to support such links for "external" test results)

Alternative idea for implementation:

make test runner do something before every test that openQA can easily notice (write a line to a serial console? display some very distinctive window?)
modify how we run tests to take a screenshot every time such thing is noticed (which will make it linked to the video timestamp)
somehow make it easy to find which screenshot is about what test (inject test name as "info" result in between?)

The value to a user, and who that user might be

Developers can easily find more details about failed tests.

andrewdavidwong commented 3 years ago

Changing milestone to "non-release," as this sounds like an improvement to our testing infrastructure rather than an enhancement that will actually be implemented in Qubes OS itself in some specific future release. Please let me know if I'm incorrect.

deeplow commented 2 years ago

Addressed partly in https://github.com/QubesOS/openqa-tests-qubesos/pull/16.

convince openQA to link it (may require extending openQA to support such links for "external" test results)

Except for this part.

QubesOS / qubes-issues