search

DUNE-DAQ / daq-release

Scripts and configuration files for the DUNE DAQ release
https://dune-daq-sw.readthedocs.io/en/latest/packages/daq-release/
2 stars 0 forks source link

Run all nightly integration tests even if one or more fail #354

Closed andrewmogan closed 5 months ago

andrewmogan commented 5 months ago

In response to Issue #351, this version of the nightly integration test workflow uses a matrix strategy to run each integration test in daqsystemtest and sets fail-fast: false so that all tests will run even if one or more fail. The overall workflow will still show a failed state even if, for example, 3/4 succeed. This also makes it easier to add more integration tests in the future.

Since the nightly actions are being shuffled around at the moment, in order to test this, I set NIGHTLY_TAG: NFD_PROD4_240304_A9 and added ref: amogan/break_fake_data_test under Checkout daqsystemtest. This checks out a branch of daqsystemtest where I've intentionally broken fake_data_producer_test.py. As desired, all tests ran, fake_data_producer_test failed, and the overall workflow status showed that it failed.

jcfreeman2 commented 5 months ago

The core functionality looks good. One suggested improvement: if I scroll down at the output of https://github.com/DUNE-DAQ/daq-release/actions/runs/8209587413, what I see is the results of the full suite of tests printed out for each test. Could we eliminate this redundancy?

andrewmogan commented 5 months ago

The redundant test summary output was caused by the xml parsing step being under the same job as the integration tests. Since the tests now run in a matrix strategy, the xml parsing was run separately for each integration test. The xml parsing step is now factored into its own job which only runs after all integration tests finish. See here.

andrewmogan commented 5 months ago

Should be fixed now. The solution was just to move the if: always() condition in the final step up in scope so that it runs even if the previous step fails. See here.

jcfreeman2 commented 5 months ago

Output in https://github.com/DUNE-DAQ/daq-release/actions/runs/8299667779 looks like what we're shooting for; approved.