quipucords / camayoc

Test automation framework that facilitates functional testing of quipucords.
https://camayoc.readthedocs.io/
GNU General Public License v3.0
5 stars 4 forks source link

Fix to call the 'dsc' command properly in tests #512

Closed ruda closed 6 months ago

ruda commented 6 months ago

When I installed the dsc CLI from RPM, I've discovered that some tests were failing because they're using qpc as the discovery command line name instead of dsc. This PR fix this misbehavior.

codecov[bot] commented 6 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 76.72%. Comparing base (75e944b) to head (c836346).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #512 +/- ## ========================================== - Coverage 76.79% 76.72% -0.07% ========================================== Files 7 7 Lines 698 696 -2 ========================================== - Hits 536 534 -2 Misses 162 162 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

mirekdlugosz commented 6 months ago

1. If you use dsc, then you need to change both executable and display_name in config. You can do that through environment variables - discovery-ci is doing that, see cli_override_envvars in utils/commands.py. As far as I am aware, there are no problems when you change both. We did consider automatically deriving display_name from executable, but that was never implemented.

2. display_name was added in #396 . At the time we changed qpc to not display certain information unless -v flag is passed, which caused a number of failures in Camayoc. There was a flurry of PRs: #396, #394, #403, #397. #396 was added with the intent of changing executable to qpc -v (which I believe we did for few weeks), and was left around because it might be useful for toolbox-based CLI. We eventually abandoned toolbox and moved to RPMs.

3. Which brings us to the current state - do we still need to separate executable and display_name? I'm not sure. I guess it may be useful, although I don't think we really benefited from that in last 10 months or so. In CI, you would have qpc/dsc installed from RPM, and both settings set to qpc or dsc, respectively. Locally, you would probably have qpc installed from git in the same virtual environment, and both settings set to qpc. I would like to hear if anyone sets up their environment differently and changes these two settings to different values.

If we decide to proceed with that change, my only request would be to leave some notes in commit description - refer to commit 7396c6860ba8d509928fe08673d7db1bec0a26b0 and GitHub PR number #396, at least. I have my doubts, but maybe it will help someone in the future.

mirekdlugosz commented 6 months ago

Summary of discussion on Slack:

ruda commented 6 months ago

To give more context, I have set in camayoc config. I'm not using the absolute path for dsc.

quipucords_cli:
  executable: "dsc"

When Camayoc assumes qpc as defaults to quipucords_cli.display_name, it results in test sources and scans failures (those who uses display_name).

ruda commented 6 months ago

Folks, to avoid going further with aggressive / big impact changes to the framework, I'm proposing to change just the tests in PR #513 and they will use the CLI executable.This is an intermediate step before changing anything regarding the display_name. I'm open to your opinions.