quipucords / camayoc

Test automation framework that facilitates functional testing of quipucords.
https://camayoc.readthedocs.io/
GNU General Public License v3.0
5 stars 4 forks source link

Add RHACS automation #461

Closed mirekdlugosz closed 11 months ago

mirekdlugosz commented 11 months ago

This adds CLI test (modeled after openshift test), and support for UI.

CLI test runs a scan and verifies a details report - checks fields and their values. Test will fail if you try to run it against empty RHACS (technically such test is valid, but I don't think it's very valuable - and I would rather keep stronger tests here). RHACS has a known problem where "max" fields are updated with a delay after cluster was registered, which means that test will likely fail until RHACS data is all processed.

UI test follows the same principles as other UI tests - just create a source, run a scan, download report, verify that report is syntactically correct. No data is validated.

$ pytest -v -ra camayoc/tests/qpc/cli/test_rhacs.py camayoc/tests/qpc/ui/test_endtoend.py --tracing retain-on-failure --browser chromium -k rhacs
camayoc/tests/qpc/ui/test_endtoend.py::test_trigger_scan[chromium-rhacs-testing-rhacs] SKIPPED (Skipped due to intermittent failure - DISCOVERY-426)                   [ 33%]
camayoc/tests/qpc/cli/test_rhacs.py::test_rhacs_data[rhacs-testing] PASSED                                                                                             [ 66%]
camayoc/tests/qpc/ui/test_endtoend.py::test_end_to_end[chromium-rhacs-testing-rhacs] PASSED                                                                            [100%]
========================================================== 2 passed, 1 skipped, 10 deselected, 5 warnings in 44.20s ==========================================================
codecov[bot] commented 11 months ago

Codecov Report

Merging #461 (22e2fc6) into main (571e59f) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #461   +/-   ##
=======================================
  Coverage   77.07%   77.07%           
=======================================
  Files           4        4           
  Lines         205      205           
=======================================
  Hits          158      158           
  Misses         47       47           

:mega: Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

ruda commented 11 months ago

Update: the Camayoc configuration have to have a dummy entry for RHACS credentials, otherwise the problem I found bellow will happens. Here is my dummy RHACS credentials:

  - name: 'rhacs'
    type: 'rhacs'
    auth_token: '123456789'    
camayoc/tests/qpc/ui/test_sources.py::test_create_delete_source[chromium-RHACSSourceFormDTO] FAILED                          [ 20%]List of previous successfull actions:

camayoc/tests/qpc/ui/test_sources.py::test_create_delete_source[chromium-NetworkSourceFormDTO] PASSED                        [ 40%]
camayoc/tests/qpc/ui/test_sources.py::test_create_delete_source[chromium-SatelliteSourceFormDTO] PASSED                      [ 60%]
camayoc/tests/qpc/ui/test_sources.py::test_create_delete_source[chromium-VCenterSourceFormDTO] PASSED                        [ 80%]
camayoc/tests/qpc/ui/test_sources.py::test_create_delete_source[chromium-AnsibleSourceFormDTO] PASSED                        [100%]
...
camayoc/tests/qpc/ui/test_sources.py:63: in create_source_dto
    credential_model = data_provider.credentials.new_one({"type": credential_type}, data_only=False)
camayoc/data_provider.py:137: in new_one
    return next(new_generator)
camayoc/data_provider.py:119: in new_many
    matching_definitions = self._select_definitions(match_criteria)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <camayoc.data_provider.ModelWorker object at 0x7fc808ef4a10>, match_criteria = {'type': 'rhacs'}

    def _select_definitions(self, match_criteria):
        matching = self._defined_models.where(**match_criteria)
        if not matching:
            msg = (
                "No data matching provided criteria. "
                "Try changing 'match_criteria' or review Camayoc configuration."
            )
>           raise NoMatchingDataDefinitionException(msg)
E           camayoc.exceptions.NoMatchingDataDefinitionException: No data matching provided criteria. Try changing 'match_criteria' or review Camayoc configuration.

camayoc/data_provider.py:42: NoMatchingDataDefinitionException
===================================================== short test summary info ======================================================
FAILED camayoc/tests/qpc/ui/test_sources.py::test_create_delete_source[chromium-RHACSSourceFormDTO] - camayoc.exceptions.NoMatchingDataDefinitionException: No data matching provided criteria. Try changing 'match_criteria' or revi...
=================================================== 1 failed, 4 passed in 11.84s ===================================================in 1.35s =========================================================