search

ansible / awx-plugins

https://awx-plugins-core.rtfd.io
Apache License 2.0
6 stars 12 forks source link

re-run failed tests triggers on failed coverage #21

Open chrismeyersfsu opened 3 months ago

chrismeyersfsu commented 3 months ago

https://github.com/ansible/awx-plugins/actions/runs/10597353901/job/29367631346?pr=17

FAIL Required test coverage of 100% not reached. Total coverage: 40.78%

It looks like the re-run failed tests with higher verbosity triggers re-running ALL tests when the coverage check fails. In the case above all the tests passed both runs.

webknjaz commented 3 months ago

So… Re-runs are made to always fail so that they don't hide flakiness. There's a || exit 1 in the command. That is intentional.

The only problematic bit I noticed are tracebacks coming from commands_post.

Any other action item I'm missing?

chrismeyersfsu commented 3 months ago

I think I'm wrong about coverage failure being the reason.

I was trying to point out that all tests pass (first screenshot) image

Then the re-run seems to run ALL the tests. I would expect it to only re-run the tests that failed above (of which there are none) (second screenshot) image

There is also an array index error but I think that may be related to coverage.

webknjaz commented 3 months ago

No, the index error is related to the last verbose tox call w/o coverage collection. I only tested the first one because there were no failures in the skeleton. So reruns weren't triggered. I fixed that already.

The reruns are generic and trigger on any errors in previous GHA steps.

The pytest rerun is invoked with --lf to pick up the failed tests but since there's none, all are executed again. I think you're right that coverage likely causes the failure of that first testing invocation.