The addresses of the vault server are now centralized into variables instead of being duplicated and spread throughout the test files. This will make it easier to specify an externally running Vault server to run tests against, furthering the separation between server prep and tests against it.
Other variables related to configuring or starting a Vault server also moved centrally.
The Vault server setup is now split between the acts of downloading and starting a local Vault server vs. running commands against the running Vault server (setting up backends, secrets, auth methods, etc.). Another change toward supporting a running Vault server that wasn't started by the test suite itself.
Vault HTTP and HTTPS tests are more clearly separated and the HTTPS tests can be optionally skipped entirely by setting a single variable. This will make for easier local setup of an external Vault server (like in a container) for those who are doing development and running tests locally, but almost certainly don't need to run the HTTPS tests.
The cert/key file locations for HTTPS vault are now located centrally in variables as well, so that they aren't strewn about.
While these are being pushed into the one defaults/main.yml file for now, the real point of this is to be able to move these operations into their own roles, so that those operations are more discrete. In addition, when we remove the need for runme.sh and have more traditional role-based tests, we can use the .gitignored integration_config.yml to control these values in different testing scenarios.
Some minor updates, like comments, text fixes, etc.
Things still to do and things learned
(these are musings mostly to remind myself)
doing HTTPS tests against an external vault server is tricky. It needed to be set up to listen with TLS, which means it needed a cert, and then for the tests to test name matching functionality, the tests need to connect to that Vault instance with a name that matches the cert; further if the cert was self-signed, we need a copy of the cert to use as a "ca" file so that the cert "chain" can be verified.
Mid-term, since the connection parameters have been moved to central code, I'd like to test connection stuff with its own set of integration tests, and not re-run every auth test x every connection test, as it will greatly reduce the number of tests needed to be run. But this doesn't solve the stickiness with setting up the Vault server.
It gets easier if we don't "test" name validation from end to end, which may be for the best: the part we actually care about is that the Ansible content correctly sends the options to hvac, from there what happens between hvac and a Vault server is largely out of our hands. That would make for a test that can be done much more easily within unit tests. There is a possibility of uncaught bugs/regressions in hvac (or even Vault) and it would be nice to catch those early in case they can be worked around but I'll have to consider whether that's worth the complexity. A middle ground might be integration tests for TLS and cert verification that have hvac talking to an httptester (the requests will fail but it should be detectable that the failures are related to not talking to a Vault server, as they will have "passed" the point of failing for TLS-related reasons).
Apart from the above, when running tests locally it'll be nice to skip all of those redundant tests since most work being done will not be touching that, and the TLS tests will still run in CI.
We have a test for ensuring that retrieval of a secret version works, but this is done by manually adding a query string to your secret path; it's not really a documented feature of the plugin. This relies on adding a second version of a secret during Vault server configuration. As I was testing I ran into the issue that with an externally running Vault server, re-running the configuration over and over (which now works without failure after #88), eventually causes that secret to hit its version limit, and version 2 of the secret (which is the hardcoded version in the tests), no longer exists. So I need to think about whether this test is actually needed/useful at all (I'm leaning no, we make no guarantee that will work), and if it is, need to complicate the secret setup a bit to ensure that there's always a version 2, or complicate the test to determine a valid version number.
There are some tests where data retrieved during the configuration stage needs to get used in the testing stage: for example, tokens are created for the token tests, AppRole auth method generates a new approle and secret ID for use in their tests. I have a flag to skip the configuration altogether (useful when your external server has already been set up and you wanna save a little speed by skipping the unnecessary configuration), but then those tests fail because they don't have the info they need anymore.
The token one can be fixed by moving those two tasks (the only setup tasks for token) into the test part themselves: they are runtime actions after all.. even though they do create a Vault object.
For AppRole, we can change from using generated IDs, to telling Vault the AppRole ID and Secret ID we want (I think), even though that's unusual. That would let us hardcode those IDs, create them in the configuration phase, and know them a priori in the test phase so other runs won't fail.
But I want to think a little more about how to best handle these, and whether to further splinter the setup/configuration type tasks into a one-time / each-time sort of paradigm (or to just push everything of that nature into the test phase.. which was my first inclination but it felt wrong looking at approle, where we'd need to not only create the approle, but also assign policies and such).
SUMMARY
More test improvements following on #88
defaults/main.ymlfile for now, the real point of this is to be able to move these operations into their own roles, so that those operations are more discrete. In addition, when we remove the need forrunme.shand have more traditional role-based tests, we can use the.gitignoredintegration_config.ymlto control these values in different testing scenarios.Things still to do and things learned
(these are musings mostly to remind myself)
hvac, from there what happens betweenhvacand a Vault server is largely out of our hands. That would make for a test that can be done much more easily within unit tests. There is a possibility of uncaught bugs/regressions inhvac(or even Vault) and it would be nice to catch those early in case they can be worked around but I'll have to consider whether that's worth the complexity. A middle ground might be integration tests for TLS and cert verification that havehvactalking to an httptester (the requests will fail but it should be detectable that the failures are related to not talking to a Vault server, as they will have "passed" the point of failing for TLS-related reasons).ISSUE TYPE
COMPONENT NAME
ADDITIONAL INFORMATION