Test setups (anything that sets up or configures Vault) are supposed to be idempotent. The purpose is to ensure that we do not specifically rely on the Vault server being destroyed and recreated every time (this might be needed during some test development, but should not be needed as a matter of course).
This was achieved in the past
Test setups can also set a marker, usually a canary value set in the root token's cubbyhole, to be used to indicate that certain things have already been set up and don't need to be set up again. However this was meant purely as an optimization, not meant to be the mechanism by which repeated test runs against a running Vault server continue to work.
In local test invocation in CI, we re-run the tests a second time against the running Vault server. This does ensure that tests can be run again, but it does not ensure that the underlying setup components are idempotent if they are skipped due to finding a canary/marker value.
I think I will add a third test run in the localenv tests, where before running them, we delete all cubbyhole values.
The tests are pretty slow already on MacOS though, so I may do this third run only on the ubuntu localenv tests.
SUMMARY
Test setups (anything that sets up or configures Vault) are supposed to be idempotent. The purpose is to ensure that we do not specifically rely on the Vault server being destroyed and recreated every time (this might be needed during some test development, but should not be needed as a matter of course).
This was achieved in the past
Test setups can also set a marker, usually a canary value set in the root token's cubbyhole, to be used to indicate that certain things have already been set up and don't need to be set up again. However this was meant purely as an optimization, not meant to be the mechanism by which repeated test runs against a running Vault server continue to work.
In local test invocation in CI, we re-run the tests a second time against the running Vault server. This does ensure that tests can be run again, but it does not ensure that the underlying setup components are idempotent if they are skipped due to finding a canary/marker value.
I think I will add a third test run in the localenv tests, where before running them, we delete all cubbyhole values.
The tests are pretty slow already on MacOS though, so I may do this third run only on the ubuntu localenv tests.
ISSUE TYPE
COMPONENT NAME
ADDITIONAL INFORMATION