Epic: New Tooling Environment

[mogul]

What we're after

Tooling environments are deployed and operated independent of the production AWS account.

Hypothesized benefit(s)/why:

• Reduce the potential for a compromise in tooling environments to affect the production environment
• Increases our ability to stand up additional tooling environments, eg for special projects
• Increases our ability to use a single tooling environment to manage multiple development, staging, or production accounts

Potential metrics

• The number of assets in the production account that are used for the tooling environment should approach zero.

---

Further context for those unfamiliar with what we're doing

Currently our tooling and deployment infrastructure all sits within the same account as the production environment. These components include:

• Master Bosh, tooling bosh, and environment bosh
• Concourse
• Ops Uaa
• Credhub
• Tooling Networking
• Platform Logging

At the end of this epic we would want to see the cloud.gov tooling and deployment solution, comprised of most of the systems above ( and others as needed), contained within its own AWS account but capable of deploying and connected to our environments in multiple accounts and regions. We would want the solution to be deployable with minimal manual bootstrapping required and have it deployable and updateable as automated as possible.

This work should happen in advance of creating and deploying new stand alone environments in their own accounts. We would incrementally deploy and prove promotion of new environments of dev and staging, before tearing down the existing dev and stage. Finally after prod has been migrated as promoted environment, then tear down existing tooling.

Security considerations

[note any potential changes to security boundaries, practices, documentation, risk]

Notes for implementers

• Work for this will likely require the refactoring and modularization into multiple repos the core Terraform for provisioning tooling infrastructure.
• Work may require the use of a hosted pipeline service like AWS codepipeline/codebuild to be used for bootstrapping automation
• Pipelines required for updating and continuous delivery of this solution should themselves be self-updating and gitops operated.
• Credentials and secrets for this solution should be maintained separately from configuration parameters and tooling created to update those credentials as needed.
• Shared VPCs across accounts, peering VPCs, or deploying a transit gateway may be required to regulate traffic flow and access for tooling. This network configuration on an organizational scale should be maintained as part of the configuration for this solution and kept in this account.

Related issues/sub-projects

• [ ] Refactor cg-provision for tooling and enviro breakout
  • [ ] Take related 'modules' and move to their own git repos, increase portability and configurability.
  • [ ] Refactor stacks to their own env repos, use configuration, and import modules from git
• [ ] New TF module for x-account networking
• [ ] New TF module for multi env DNS and multiple hosted sub zones
• [ ] Refactor and pattern/template pipelines for promotion based on fixed version
  • [ ] Create templates and generator tooling to build an enviro based on stored metadata from within another 'controller' pipeline
  • [ ] Create 'controller' pipeline
  • [ ] Create 'state' repo(s) and metadata for generated pipelines and state of deployed env (versions)
• [ ] Create/Update and self deploy pipeline for tooling env after bootstrapping using 'controller' pipeline
• [ ] Deploy new tooling env (bosh and components) to new account
• [ ] supporting tooling/scripts and plan to migrate (future) data from existing tooling
• [ ] supporting tooling/scripts for bootstrapping tooling environment
• [ ] supporting tooling/scripts for backup of tooling environment

### Tasks

### Tasks
- [ ] https://github.com/cloud-gov/product/issues/2525
- [ ] https://github.com/cloud-gov/product/issues/2565
- [ ] https://github.com/cloud-gov/product/issues/2576
- [ ] https://github.com/cloud-gov/product/issues/2574
- [ ] https://github.com/cloud-gov/product/issues/2527
- [ ] https://github.com/cloud-gov/product/issues/2578
- [ ] https://github.com/cloud-gov/product/issues/2571
- [ ] https://github.com/cloud-gov/product/issues/2570
- [ ] https://github.com/cloud-gov/product/issues/2575
- [ ] https://github.com/cloud-gov/product/issues/2573
- [ ] https://github.com/cloud-gov/product/issues/2572
- [ ] https://github.com/cloud-gov/product/issues/2677

[mogul]

Can anyone comment as to the overall status of this epic, given work in the most recent quarter?

[pburkholder]

Do we have more than 🦗 on this?