quipucords / camayoc

Test automation framework that facilitates functional testing of quipucords.
https://camayoc.readthedocs.io/
GNU General Public License v3.0
5 stars 4 forks source link

Introduce UIv2 support #541

Closed mirekdlugosz closed 1 month ago

mirekdlugosz commented 1 month ago

Changes needed in Camayoc to tun tests against UIv2.

This is being developed against https://github.com/quipucords/quipucords-ui/pull/463 , which is in draft state. The current plan is:

  1. Have enough changes in Camayoc to have at least one of each UI tests passing when running against new UI. That should identify most of changes needed in quipucords-ui for Camayoc to find elements in ergonomic way.
  2. Merge quipucords-ui PR. Open this PR for merging
  3. Merge this PR
  4. Create nightly pipeline that executes Camayoc with UIv2 enabled against quipucords-ui main. There are going to be failures, which will be tackled in future PRs.
  5. ???
  6. Profit. I mean, release new UI. Assuming pipeline is green.

There are multiple commits, as I try to have one commit per area of interest. We can squash later if that is desired.

How to run:

  1. Obtain the quipucords-ui container image with https://github.com/quipucords/quipucords-ui/pull/463 . You might need to build it locally
  2. Start quipucords server, using above container image for quipucords-app. Make sure that Quipucords is using UIv2.
  3. Modify Camayoc config to enable use_uiv2 setting. Or run export DYNACONF_camayoc__use_uiv2=True.
  4. Start UI tests:
    pytest -v -s 'camayoc/tests/qpc/ui/test_login.py' --tracing retain-on-failure --browser chromium
codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 77.90%. Comparing base (5bb28de) to head (8bcc417). Report is 6 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #541 +/- ## ======================================= Coverage 77.90% 77.90% ======================================= Files 7 7 Lines 688 688 ======================================= Hits 536 536 Misses 152 152 ``` | [Flag](https://app.codecov.io/gh/quipucords/camayoc/pull/541/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=quipucords) | Coverage Δ | | |---|---|---| | [](https://app.codecov.io/gh/quipucords/camayoc/pull/541/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=quipucords) | `77.90% <ø> (?)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=quipucords#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

mirekdlugosz commented 1 month ago

This is now ready for review and merge.

Local run results against UI v2 with quipucords-ui #463:

=================================================================== short test summary info ===================================================================
FAILED camayoc/tests/qpc/ui/test_endtoend.py::test_end_to_end[chromium-middleware-testlab-network] - playwright._impl._errors.TimeoutError: Page.fill: Timeout 30000ms exceeded.
FAILED camayoc/tests/qpc/ui/test_endtoend.py::test_end_to_end[chromium-testlab-B-network] - playwright._impl._errors.TimeoutError: Page.fill: Timeout 30000ms exceeded.
FAILED camayoc/tests/qpc/ui/test_endtoend.py::test_end_to_end[chromium-testlab-A-network] - playwright._impl._errors.TimeoutError: Page.fill: Timeout 30000ms exceeded.
FAILED camayoc/tests/qpc/ui/test_credentials.py::test_create_delete_credential[chromium-SSHNetworkCredentialFormDTO] - playwright._impl._errors.TimeoutError: Page.fill: Timeout 30000ms exceeded.
===================================================== 4 failed, 22 passed, 8 skipped in 577.05s (0:09:37) =====================================================

All failing tests try to run Network scan while using SSH. That fails because UI moved from a path to SSH file to SSH file content. Adding support for that will require larger changes in framework and I would prefer to tackle it in another PR.