quipucords / camayoc

Test automation framework that facilitates functional testing of quipucords.
https://camayoc.readthedocs.io/
GNU General Public License v3.0
5 stars 4 forks source link

Port CLI report tests to scans fixture #521

Closed mirekdlugosz closed 4 months ago

mirekdlugosz commented 4 months ago

Move away from setup_reports_prerequisites autouse fixture to scans fixture, which is shared across CLI and API tests.

The main reason for this is consistency. There is a slight difference in functionality - while before these tests were run only for network scans, and each scan had a single source, now they run for network / vcenter / satellite scan, and each scan might have more than single source (depending on configuration file).

This should also speed up entire suite execution time. Since old way cached scan results in file, but not across entire suite, in total it added about 160 seconds to full run time. However, this benefit is limited to running the full suite, or individual tests from the file.

This partly resolves DISCOVERY-492

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 78.13%. Comparing base (e3f6669) to head (61ca3f7).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #521 +/- ## ======================================= Coverage 78.13% 78.13% ======================================= Files 7 7 Lines 677 677 ======================================= Hits 529 529 Misses 148 148 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

ruda commented 4 months ago

I ran the Camayoc tests against quipucords upstream and I found two errors and I think it may not be related to this change.

FAILED camayoc/tests/qpc/cli/test_reports.py::test_merge_report[json-file] - AssertionError: Checking files for valid json details report. ['/tmp/b9bne5ge/testlab-4e51e49d-5773-4892-aa00-bc6f68b7fd71.json', '/tmp/b9bne5ge/middleware-testlab-e1b00dde-56c4-482c-95a4-9710c11380a8.json']
FAILED camayoc/tests/qpc/cli/test_reports.py::test_upload_report - AssertionError: Checking /tmp/pytest-of-rmoura/pytest-0/test_upload_report0/original-report.json for valid JSON details report.
ruda commented 4 months ago

I repeated the test against main and I was able to reproduce the issue, so it didn't happen by your changes.