Open mrgrain opened 3 weeks ago
Would also be neat if it could keep statistics about wait time, test run time, failure rates etc.
And if we could request accounts with specific setups, like 2 accts with cross account trust, or manually prepared certificates.
I would like the service to direct the client as much as possible. So it will also hand out Role ARNs to assume, and other parameters that are necessary for the client to work properly (pre-existing resources names, unique prefixes, etc). Ideally clients only need to follow instructions, and operate on as few assumptions of the target environments as possible
And if we could request accounts with specific setups, like 2 accts with cross account trust, or manually prepared certificates.
I would like the service to direct the client as much as possible. So it will also hand out Role ARNs to assume, and other parameters that are necessary for the client to work properly. Ideally clients only need to follow instructions, and operate on as few assumptions of the target environments as possible
Yes, but. We also need to provision for testing of bootstrapping. But I agree that for non-bootrapping test cases, pre-bootstrapped environments would be desirable.
Sure. Maybe then we just ask for 2 accounts and do the --trust
bootstrapping as part of the test, that works too. It's probably better!
Requests for the other hand-provisioned crap still stands though 😉 .
The example I always go to is certificates and Hosted Zone IDs. I guess that doesn't apply to CLI tests yet... but (a) we could use the same coordinator for automating construct library tests and (b) I'm sure there will be something manual at some point.
It would great if this service could also be used in local development. This will make writing and developing complex tests easier, which means we will do more of them.
Perhaps we can create a dedicated pool for local dev usage.
Maybe then we just ask for 2 accounts and do the --trust bootstrapping as part of the test, that works too. It's probably better!
Yes, I like that better.
Overview
Our integ tests need to run in an exclusive environment. Currently we are achieving this roughly by two ways:
However this model doesn't scale well. We should keep the Sandbox account boundary because we won't have that many different sandboxes and setting up accounts as a one-off operation is relatively cheap.
Instead of region designations, we should however have a service that reserves and assigns a an account and region pair to a test case. Some test cases (like cross-account deployments #31934) might reserver multiple pairs. The service can also ensure environments are regularly cleaned up from any accidentally left-over resources.
Requirements
A new service an integration test can request exclusive, pre-configured environments to execute tests in.