Draft: Skip some of credentials and sources tests if sshkeyfile is missing in configuration

[ruda]

I was working with a minimal configuration file for Camayoc and notice that if the sshkeyfile in any credential configuration is missing, then some of the tests will fail with a misleading message.

I'm creating this PR to skip some of credentials and sources tests if sshkeyfile is missing in configuration.

Example.

Before:

FAILED camayoc/tests/qpc/cli/test_credentials.py::test_add_with_username_sshkeyfile - camayoc.exceptions.NoMatchingDataDefinitionException: No data matching provided criteria. Try changing 'match_criteria' or revi...
FAILED camayoc/tests/qpc/cli/test_credentials.py::test_add_with_username_sshkeyfile_become_password - camayoc.exceptions.NoMatchingDataDefinitionException: No data matching provided criteria. Try changing 'match_criteria' or revi...
FAILED camayoc/tests/qpc/cli/test_credentials.py::test_edit_sshkeyfile_negative - camayoc.exceptions.NoMatchingDataDefinitionException: No data matching provided criteria. Try changing 'match_criteria' or revi... ...

After:

SKIPPED [1] camayoc/tests/qpc/cli/test_credentials.py:140: Make sure you have at least one network credential with sshkeyfile on the config file to run this test. 
SKIPPED [1] camayoc/tests/qpc/cli/test_credentials.py:178: Make sure you have at least one network credential with sshkeyfile on the config file to run this test. 
SKIPPED [1] camayoc/tests/qpc/cli/test_credentials.py:376: Make sure you have at least one network credential with sshkeyfile on the config file to run this test. ...
SKIPPED [3] camayoc/tests/qpc/api/v1/sources/test_network_sources.py:79: Make sure you have at least one network credential with sshkeyfile on the config file to run this test. 
SKIPPED [4] camayoc/tests/qpc/api/v1/sources/test_network_sources.py:115: Make sure you have at least one network credential with sshkeyfile on the config file to run this test.

[codecov[bot]]

Codecov Report

Merging #435 (ea0f293) into main (eed0bd2) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #435   +/-   ##
=======================================
  Coverage   79.11%   79.11%           
=======================================
  Files           5        5           
  Lines         249      249           
=======================================
  Hits          197      197           
  Misses         52       52           

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[mirekdlugosz]

This is really a problem only for a single test, test_update_password_to_sshkeyfile, but I'm not a fan how it calls skip in the middle of a test, when part of a setup was already done.

I feel the cleaner solution would be to use pytest.marker.skipif decorator with a function. Function would create new DataProvider instance (unfortunately you can't use fixture) and call data_provider.credentials.new_one({"type": "network", "sshkeyfile": Table.is_not_null()}, data_only=True) (this ensures no credential is created on server side). If that call raised an exception, then return True. Otherwise, return False and let the test run.

Alternatively, a function could import settings and iterate over credentials to check if there is any with ssh key definition.

I wonder what is the level of effort needed to revert this and use decorator instead? If it's half a day of work, then I guess we should stick to what we have. Maybe only move data_provider call to beginning of setup in test_update_password_to_sshkeyfile.

[ruda]

Yeah, by using the decorator it will be a better solution. I going to keep this PR, until I have another one with the improved implementation.