search

uptane / aktualizr

C++ Uptane Client
Mozilla Public License 2.0
15 stars 15 forks source link

test_treehub_failure always broken? #45

Closed cajun-rat closed 2 years ago

cajun-rat commented 2 years ago

As part of my War on Flaky Tests™ I've been looking into what is causing test_treehub_failure to fail.

This can be reproduced with ctest --output-on-failure -R test_treehub_failure. I think it fails reliably--i.e. it isn't flaky, it is just broken :)

I've tracked it down to test_treehub_update_after_image_download_failure() in test_treehub_failure.py. The infinite redirect tests in RedirectHandler are causing the problem. The other test cases such as DownloadInterruptionHandler fail a limited number of times before succeeding, whereas RedirectHandler will trigger a chain of 1000x 301 redirects. When this happens Aktualizr correctly gives up the OTA update and reports a failure, however the test is expecting the overall OTA update to succeed.

A quick fix is git revert e47b6650500ad80999f83eef3ba78afb25314493 which will back out the change that enabled that test case. I don't have a solid handle on the Python meta-programming magic in test_fixtures.py, or I would try something fancier...

After this I think is is just test_ip_secondary_rotation :)

cajun-rat commented 2 years ago

The tests pass on my machine, and the last couple of PRs had clean CI runs.