tests - config & auth separation

SUMMARY

Continuing on the work in #106 , we can build on that to start to pull apart the tests.

Changes:

move (initial) Vault configuration (kv secrets and policies) to its own setup target
remove the gen engine (fixes: #89 )
move the none auth method to its own target
add a GitHub action that pulls the ansible-test docker images in a separate step
move connection testing to its own target, with custom test plugin

That last change alone makes the tests much faster, because we're severely reducing the number of (duplicate) tests being run.

The "strategy" previously was to test connection options by running all the auth tests, once for each set of connection options under test, so this multiplied quickly.

Running the full set of integration tests now runs in under a minute locally in docker (with the docker localenv requirements already set up beforehand), and a significant amount of that time is ansible-test --docker setup/teardown (not that it's a lot of time, just that it's now significant compared to the test time).

Full CI (when GitHub instance scheduling is behaving) has gone down from lows of 18-20m, to around 11m.

Pulling the test images in a separate step also showcases how slow it is to pull from quay.io, and it lets us see the time for running the tests themselves more accurately.

I expect to further reduce that in another PR by testing multiple versions of Vault in the same GitHub "Job", essentially launching several Vault containers in the same job, and changing the tests to be able to handle running N times (once for each version/"target"), or perhaps just re-running ansible-test and the localenv once for Vault version (which would be slower, but simpler to implement).

I may also consider running all the Python versions in the same job as well, because the isolation for python happens in the test container anyway, and there's little reason to spawn new GHA VMs and set them up for that purpose.

The point of that in reducing CI time is twofold:

there is significant overhead in waiting for GitHub to schedule a new instance, and then to set it up to actually run a test; being able to fold the tests against other Vault versions into the same environment (same Ansible & python version or maybe even all python versions) makes sense, and comes at a much smaller cost (in time) than launching a whole new job
low concurrency limits on GitHub actions mean the more jobs we have (with all their overhead), the longer we have to wait for them to even start. Reducing this will greatly reduce total CI time

Splitting out the none auth method was more of a proof of concept: that we can (even now without having moved it to module utils) have auth methods in their own targets. I moved that because it's by far the simplest one.

I expect to move others as well, though I would like to coincide it at least roughly with each auth method's move to its own shared class. Each of those targets will include its own Vault server config (all of which right now is still within the lookup plugin tests along with the other auth methods).

But we're getting closer and closer to pulling these apart.

ansible-collections / community.hashi_vault

tests - config & auth separation #111

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

Codecov Report