Test environment reservation service

[mrgrain]

Overview

Our integ tests need to run in an exclusive environment. Currently we are achieving this roughly by two ways:

• use of sandbox accounts to establish strong boundaries (e.g. canaries vs test pipelines)
• designation of regions per test to separate different tests within the same sandbox

However this model doesn't scale well. We should keep the Sandbox account boundary because we won't have that many different sandboxes and setting up accounts as a one-off operation is relatively cheap.

Instead of region designations, we should however have a service that reserves and assigns a an account and region pair to a test case. Some test cases (like cross-account deployments #31934) might reserver multiple pairs. The service can also ensure environments are regularly cleaned up from any accidentally left-over resources.

Implementation / Features

A new service an integration test can request exclusive, pre-configured environments to execute tests in.

• MUST support exclusive reservations for pairs of account/region from a pool
• MUST support different pools to safely separate out different streams (e.g. release, canaries, regular tests)
• MUST support requesting combinations for cross-account deployment use cased
• MUST provide the requesting client with required credentials (i.e. role arn) to execute tests
• MUST support metrics about wait time, test run time, failure rates etc.
• SHOULD support bootstrapping of envs for a given reservation, e.g. configuring 2 accounts with cross account trust
• SHOULD support clearing of environments from accidentally left-over resources
• SHOULD support metrics on accidentally left-over resources and the reservations that caused that

[rix0rrr]

Would also be neat if it could keep statistics about wait time, test run time, failure rates etc.

And if we could request accounts with specific setups, like 2 accts with cross account trust, or manually prepared certificates.

[rix0rrr]

I would like the service to direct the client as much as possible. So it will also hand out Role ARNs to assume, and other parameters that are necessary for the client to work properly. Ideally clients only need to follow instructions, and operate on as few assumptions of the target environments as possible

[mrgrain]

And if we could request accounts with specific setups, like 2 accts with cross account trust, or manually prepared certificates.

I would like the service to direct the client as much as possible. So it will also hand out Role ARNs to assume, and other parameters that are necessary for the client to work properly. Ideally clients only need to follow instructions, and operate on as few assumptions of the target environments as possible

Yes, but. We also need to provision for testing of bootstrapping. But I agree that for non-bootrapping test cases, pre-bootstrapped environments would be desirable.

[rix0rrr]

Sure. Maybe then we just ask for 2 accounts and do the --trust bootstrapping as part of the test, that works too. It's probably better!

Requests for the other hand-provisioned crap still stands though 😉

[iliapolo]

It would great if this service could also be used in local development. This will make writing and developing complex tests easier, which means we will do more of them.

Perhaps we can create a dedicated pool for local dev usage.

[iliapolo]

Maybe then we just ask for 2 accounts and do the --trust bootstrapping as part of the test, that works too. It's probably better!

Yes, I like that better.