search

quipucords / camayoc

Test automation framework that facilitates functional testing of quipucords.
https://camayoc.readthedocs.io/
GNU General Public License v3.0
5 stars 4 forks source link

Enable scan tests #412

Closed mirekdlugosz closed 1 year ago

mirekdlugosz commented 1 year ago

This PR enables reports and scan / scanjobs tests. I tried to make commits relatively complete, so it's best to review things commit-by-commit.

It depends on config changes introduced in discovery-ci PR #7.

Most changes revolve around using DataProvider and untangling dependencies between functions and fixtures. I decided to not port tests that are marked as skipped - I ran them before starting this work, and it seems that referenced issues are still not solved.

Performance of current version is not great, as most tests run scans they need - this leads to servers being scanned multiple times. I think this is something we can try to tackle later on.

Test results are below. There's one failure, but that tests seems flaky - it usually fails, but I saw it passing few times. Another thing to tackle later.

$ pytest -v camayoc/tests/qpc/api/v1/reports/test_reports.py camayoc/tests/qpc/api/v1/scanjobs/ camayoc/tests/qpc/cli/test_reports.py camayoc/tests/qpc/cli/test_scanjobs.py
camayoc/tests/qpc/api/v1/reports/test_reports.py::test_report_content_consistency PASSED                                                                               [  2%]
camayoc/tests/qpc/api/v1/reports/test_reports.py::test_merge_reports_from_scanjob PASSED                                                                               [  4%]
camayoc/tests/qpc/api/v1/reports/test_reports.py::test_merge_reports_negative PASSED                                                                                   [  6%]
camayoc/tests/qpc/api/v1/reports/test_reports.py::test_products_found_deployment_report[testlab] SKIPPED (Skipped until Middleware are handled by Camayoc)             [  8%]
camayoc/tests/qpc/api/v1/reports/test_reports.py::test_products_found_deployment_report[VCenterOnly] SKIPPED (Skipped until Middleware are handled by Camayoc)         [ 10%]
camayoc/tests/qpc/api/v1/reports/test_reports.py::test_products_found_deployment_report[Sat6] SKIPPED (Skipped until Middleware are handled by Camayoc)                [ 12%]
camayoc/tests/qpc/api/v1/reports/test_reports.py::test_OS_found_deployment_report[testlab] PASSED                                                                      [ 14%]
camayoc/tests/qpc/api/v1/reports/test_reports.py::test_OS_found_deployment_report[VCenterOnly] PASSED                                                                  [ 17%]
camayoc/tests/qpc/api/v1/reports/test_reports.py::test_OS_found_deployment_report[Sat6] PASSED                                                                         [ 19%]
camayoc/tests/qpc/api/v1/scanjobs/test_pause_cancel_restart.py::test_pause_cancel[vcenter] SKIPPED (Test is flaky. Skipping until Quipucords Issue #2040 resoloved)    [ 21%]
camayoc/tests/qpc/api/v1/scanjobs/test_pause_cancel_restart.py::test_pause_cancel[network] SKIPPED (Test is flaky. Skipping until Quipucords Issue #2040 resoloved)    [ 23%]
camayoc/tests/qpc/api/v1/scanjobs/test_pause_cancel_restart.py::test_pause_cancel[satellite] SKIPPED (Test is flaky. Skipping until Quipucords Issue #2040 resoloved)  [ 25%]
camayoc/tests/qpc/api/v1/scanjobs/test_pause_cancel_restart.py::test_pause_cancel[openshift] SKIPPED (Test is flaky. Skipping until Quipucords Issue #2040 resoloved)  [ 27%]
camayoc/tests/qpc/api/v1/scanjobs/test_pause_cancel_restart.py::test_restart[vcenter] SKIPPED (Test is flaky. Skipping until Quipucords Issue #2040 resoloved)         [ 29%]
camayoc/tests/qpc/api/v1/scanjobs/test_pause_cancel_restart.py::test_restart[network] SKIPPED (Test is flaky. Skipping until Quipucords Issue #2040 resoloved)         [ 31%]
camayoc/tests/qpc/api/v1/scanjobs/test_pause_cancel_restart.py::test_restart[satellite] SKIPPED (Test is flaky. Skipping until Quipucords Issue #2040 resoloved)       [ 34%]
camayoc/tests/qpc/api/v1/scanjobs/test_pause_cancel_restart.py::test_restart[openshift] SKIPPED (Test is flaky. Skipping until Quipucords Issue #2040 resoloved)       [ 36%]
camayoc/tests/qpc/api/v1/scanjobs/test_run_scanjobs.py::test_scan_complete[testlab] SKIPPED (Test is flaky. Skipping until Quipucords Issue #2040 resoloved)           [ 38%]
camayoc/tests/qpc/api/v1/scanjobs/test_run_scanjobs.py::test_scan_complete[VCenterOnly] SKIPPED (Test is flaky. Skipping until Quipucords Issue #2040 resoloved)       [ 40%]
camayoc/tests/qpc/api/v1/scanjobs/test_run_scanjobs.py::test_scan_complete[Sat6] SKIPPED (Test is flaky. Skipping until Quipucords Issue #2040 resoloved)              [ 42%]
camayoc/tests/qpc/api/v1/scanjobs/test_run_scanjobs.py::test_scan_task_results[testlab] PASSED                                                                         [ 44%]
camayoc/tests/qpc/api/v1/scanjobs/test_run_scanjobs.py::test_scan_task_results[VCenterOnly] PASSED                                                                     [ 46%]
camayoc/tests/qpc/api/v1/scanjobs/test_run_scanjobs.py::test_scan_task_results[Sat6] PASSED                                                                            [ 48%]
camayoc/tests/qpc/api/v1/scanjobs/test_run_scanjobs.py::test_disabled_optional_products_facts FAILED                                                                   [ 51%]
camayoc/tests/qpc/api/v1/scanjobs/test_run_scanjobs.py::test_disabled_optional_products PASSED                                                                         [ 53%]
camayoc/tests/qpc/api/v1/scanjobs/test_run_scanjobs.py::test_enabled_extended_product_search_facts PASSED                                                              [ 55%]
camayoc/tests/qpc/api/v1/scanjobs/test_run_scanjobs.py::test_enabled_extended_product_search PASSED                                                                    [ 57%]
camayoc/tests/qpc/cli/test_reports.py::test_deployments_report[csv-report] PASSED                                                                                      [ 59%]
camayoc/tests/qpc/cli/test_reports.py::test_deployments_report[csv-scan-job] PASSED                                                                                    [ 61%]
camayoc/tests/qpc/cli/test_reports.py::test_deployments_report[json-report] PASSED                                                                                     [ 63%]
camayoc/tests/qpc/cli/test_reports.py::test_deployments_report[json-scan-job] PASSED                                                                                   [ 65%]
camayoc/tests/qpc/cli/test_reports.py::test_detail_report[csv-report] PASSED                                                                                           [ 68%]
camayoc/tests/qpc/cli/test_reports.py::test_detail_report[csv-scan-job] PASSED                                                                                         [ 70%]
camayoc/tests/qpc/cli/test_reports.py::test_detail_report[json-report] PASSED                                                                                          [ 72%]
camayoc/tests/qpc/cli/test_reports.py::test_detail_report[json-scan-job] PASSED                                                                                        [ 74%]
camayoc/tests/qpc/cli/test_reports.py::test_merge_report[report] PASSED                                                                                                [ 76%]
camayoc/tests/qpc/cli/test_reports.py::test_merge_report[scan-job] PASSED                                                                                              [ 78%]
camayoc/tests/qpc/cli/test_reports.py::test_merge_report[json-file] PASSED                                                                                             [ 80%]
camayoc/tests/qpc/cli/test_reports.py::test_download_report[report] PASSED                                                                                             [ 82%]
camayoc/tests/qpc/cli/test_reports.py::test_download_report[scan-job] PASSED                                                                                           [ 85%]
camayoc/tests/qpc/cli/test_scanjobs.py::test_scanjob PASSED                                                                                                            [ 87%]
camayoc/tests/qpc/cli/test_scanjobs.py::test_scanjob_with_multiple_sources PASSED                                                                                      [ 89%]
camayoc/tests/qpc/cli/test_scanjobs.py::test_scanjob_with_disabled_products SKIPPED (Skipped until Quipucords Issue #2038 us resolved)                                 [ 91%]
camayoc/tests/qpc/cli/test_scanjobs.py::test_scanjob_with_enabled_extended_products PASSED                                                                             [ 93%]
camayoc/tests/qpc/cli/test_scanjobs.py::test_scanjob_restart SKIPPED (Skipping until Quipucords Issue #2040 resoloved)                                                 [ 95%]
camayoc/tests/qpc/cli/test_scanjobs.py::test_scanjob_cancel PASSED                                                                                                     [ 97%]
camayoc/tests/qpc/cli/test_scanjobs.py::test_scanjob_cancel_paused SKIPPED (Skipping until Quipucords Issue #2040 resoloved)                                           [100%]
====================================================== 1 failed, 29 passed, 17 skipped, 1 warning in 1325.72s (0:22:05) ======================================================
codecov[bot] commented 1 year ago

Codecov Report

Merging #412 (438dd90) into main (bc61228) will decrease coverage by 2.11%. The diff coverage is 40.00%.

@@            Coverage Diff             @@
##             main     #412      +/-   ##
==========================================
- Coverage   81.22%   79.11%   -2.11%     
==========================================
  Files           5        5              
  Lines         245      249       +4     
==========================================
- Hits          199      197       -2     
- Misses         46       52       +6
Impacted Files Coverage Δ
camayoc/utils.py 71.87% <14.28%> (-8.78%) :arrow_down:
camayoc/api.py 69.47% <100.00%> (+0.65%) :arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more