tests - massive test refactor

[briantist]

SUMMARY

This is a big refactor of the integration tests that I've been trying to do in smaller bits and pieces for many months. Some of the more difficult and entangled aspects were not possible or practical to do in smaller increments, and I failed several times trying in previous PRs. So as much as I prefer avoid such giant changes, this was an area I found difficult to make meaningful progress otherwise.

Changes

• The tests no longer run as runme.sh tests, and now use the "standard" role-based mechanism.
• The parts of the tests responsible for setting up the proxy server and the Vault server have been completely decoupled from the lookup tests and split into their own setup_ roles.
• The integration tests now expect that addresses for Vault and a proxy server and any future external dependencies will be passed in via vars, so setting these things integration_config.yml is now required.
• The "legacy" all-in-one type of setup where the tests download, extract, and run a Vault server (and install, configure, run a proxy server), is still available for now, but it must be requested specifically (opt-in).
• A sample integration_config.yml.sample file is included. It can be copied directly to integration_config.yml to go right back to the old style all-in-one setup. This will prove to be less and less viable for running all tests as more targets get added, because that process will be repeated for every target, so it's best for one-time/infrequent contributors who will run only a single target locally.
• A concept of a "localenv" setup role is introduced, which are basically roles/playbooks that are intended to setup the dependencies outside of the test environment, and generate an integration_config that you can use to then run the tests. That gets you some re-use of those things, simplifying and speeding up local test runs.
• Included is a "docker" localenv that uses docker-compose to run the Vault server and proxy server in containers. I expect this to be the most used/most popular option.
• The GitHub workflows have been updated to use a "gha" localenv setup, which is just a special case of the docker one that uses pre-generated certificates and skips the community.docker tasks (using the already included docker-compose CLI), in order to remove a dependency on community.crypto and community.docker in the "main" CI test runs, and avoid the time taken to run those tasks.
• New test runs have been added to test the "legacy" modes of operation (since those aren't used in main CI anymore), and to test the docker localenv, as well as running ansible-test with --docker and with --venv. These tests run on ubuntu and macos, to try to ensure that we're providing a good experience for contributors.
• The addition of the above tests led to several fixes of the tests to get them running correctly, especially on MacOS.
• Getting docker installed on MacOS runners is a PITA; I combined the set of shell steps into a GitHub action.
• The local tests do still have external collection dependencies, now depending on community.crypto, community.docker, and community.general (macos only). Since installing from Galaxy can still be error prone, I've chosen to continue installing directly from GitHub, but I've wrapped this up into a neat GitHub Action as well, so it's quite clean in the workflow.
• New guides have been added to the docsite, around contributions and local testing. Those will continue to be expanded over time.

What's missing?

There's a few things that are NOT done yet in this PR

• The actual configuration of a running Vault server (setting up secrets, enabling and configuring backends) is still in the lookup target. Moving that out will be the next step, but I want to get this localenv stuff and the documentation out because it's already useful as is (and this PR is already too big).
• No additional stuff that could be moved to its own target (like connection tests) has been moved yet. This will probably be the next thing as soon as the Vault runtime config mentioned above is moved out. Once that's done I think we're well set to continue moving functionality out of the lookup plugin and into module_utils, and subsequently separating out integration testing for it. And that will lead straight into being able to make new plugins and modules.

Fixes #19 (that issue is barely relevant anymore, but the tests that now run on MacOS did surface a few issues that are now fixed)

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

[codecov[bot]]

Codecov Report

Merging #106 (799d195) into main (fc80958) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #106   +/-   ##
=======================================
  Coverage   83.52%   83.52%           
=======================================
  Files          16       16           
  Lines         892      892           
  Branches       87       87           
=======================================
  Hits          745      745           
  Misses        129      129           
  Partials       18       18           

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update fc80958...799d195. Read the comment docs.

[briantist]

@acozine thank you so much for the review! Your comments sparked a whole set of updates to the docs, in both formatting and content. If you have some time I'd really appreciate another look.

I'm also attaching rendered HTML versions of each of the pages, though the :ref: links and other content may not work right, unless you plop them into a pre-built coresite docs/docsite/_build/html directory. So I'm also including a ZIP of the full docsite build which might be easier.

just my pages (html.zip) whole Ansible docsite (full.zip)