quipucords / camayoc

Test automation framework that facilitates functional testing of quipucords.
https://camayoc.readthedocs.io/
GNU General Public License v3.0
5 stars 4 forks source link

DISCOVERY-621: Mark slow tests #526

Closed mirekdlugosz closed 3 months ago

mirekdlugosz commented 3 months ago

Using data from Jenkins, add slow marker to all tests that on average take at least 30 seconds to complete. There are 11 tests like that. If we put threshold at 15 seconds, there would be 18 tests marked.

It's important to remember that data is obtained from complete test suite runs, and so it differs from times we would obtain when running tests individually. Good example is camayoc/tests/qpc/api/v1/reports/test_reports.py - only first two tests gets marked, because all subsequent tests would use results cached in scans fixture, and finish much faster. But if you were to run camayoc/tests/qpc/api/v1/reports/test_reports.py::test_installed_products_deployment_report individually, it would likely take long time to finish, as it would have to wait for scan to complete.

Another approach we might take is just adding slow marker automatically to tests that use scans fixture, leaving explicit mark to few tests that are slow due to other reasons.

codecov[bot] commented 3 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 78.33%. Comparing base (4a03602) to head (200630b).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #526 +/- ## ======================================= Coverage 78.33% 78.33% ======================================= Files 7 7 Lines 683 683 ======================================= Hits 535 535 Misses 148 148 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.