search

quipucords / camayoc

Test automation framework that facilitates functional testing of quipucords.
https://camayoc.readthedocs.io/
GNU General Public License v3.0
5 stars 4 forks source link

Proper end to end tests utilizing Playwright #370

Closed mirekdlugosz closed 2 years ago

mirekdlugosz commented 2 years ago

This PR includes / builds upon #365 . Story is similar to #369 - I can include this in #365 , or we can close #365 and continue discussion here, or I can rebase this once #365 is merged.

This is parallel to #369 and one should nicely rebase on top of another. I didn't include one in another to indicate that this is largely independent work.

This introduces proper end-to-end UI tests. For each source type (Network, Satellite and vCenter), it executes following path:

  1. Login
  2. Create credential
  3. Create source
  4. Trigger scan
  5. Download report
  6. Logout
  7. Run assertions on report

Created data is later removed using API, so we should be able to run test repeatedly without leaving too much rubbish behind.

Input data for forms may be specified inside function. I have done this for Network source and credential, based on data available on OpenStack instance. I did not find reference values for Satellite and vCenter, so these were left out. When running tests, use -k network to only run test that should pass.

Report assertions are relatively weak - I only check that file can be extracted, that all files inside have non-zero size and that their SHA256 sums match what is specified in report file. We probably should focus more on reports content, but (1) reports are the same no matter the interface, and we already have some reports tests in CLI and API; (2) report helpers I have found are not easy to use, and at this point I am not in position to fully refactor them.

codecov[bot] commented 2 years ago

Codecov Report

Merging #370 (50b5210) into master (f4bb8a7) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #370   +/-   ##
=======================================
  Coverage   79.82%   79.82%           
=======================================
  Files           5        5           
  Lines         228      228           
=======================================
  Hits          182      182           
  Misses         46       46

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update f4bb8a7...50b5210. Read the comment docs.

ruda commented 2 years ago

Thanks for another great contribution. LGTM!