Plan and estimate adding per-PR web validator deployments

[themightychris]

Outline an approach and estimate the number of hours for one or both scenarios:

• Pushes to master deploy client+server to a pre-production staging environment (while production shifts to only being updated when releases are created) spun out into #1486 and #1487
• Select PR environments (either based on files changes or a label/tag/prefix on the PR) get deployed to ephemeral environments

In either case, the same GCS bucket that exists already can be used, there doesn't appear to be any reason to isolate the bucket storage by environment.

[themightychris]

A related piece of work which depends on this but may be scoped separately is updating the production build process triggered by releases to also generate a rules.json file and update it as part of the release artifacts.

[lottspot]

Description of Approaches

Approach-agnostic details

Both approaches will fundamentally require the implementation of three functions:

1. Provision environment-specific deployment target in GCP
2. Use web pipeline to perform environment-specific build
3. Use web pipeline to perform environment-specific deployment

In either case, the same GCS bucket that exists already can be used, there doesn't appear to be any reason to isolate the bucket storage by environment.

There are four GCS buckets associated with the application

1. Terraform state
2. Web client assets
3. Web service uploads
4. Web service reports

Buckets 1-3 MUST be recreated for each new environment due to various collision factors. Bucket 4 may be reused between environments.

The name of bucket 3 is currently hard coded into the service; this must be changed before any parallel environment can be deployed. The name of bucklet 4 is also currently hard coded into the client and service, but this does not present the same problem due to its ability to be reused without any collisions.

Approach 1

Pushes to master deploy client+server to a pre-production staging environment (while production shifts to only being updated when releases are created)

This approach would involve managing a long-lived staging environment alongside the production environment within the gtfs-infrastructure repository. This would require:

1. Hard coded inputs
   • tfbackend/state bucket for staging environment
   • tfvars for staging deployment target
2. Merge-time actions:
   1. Provision changes to staging environment

Separately, the web pipeline in the validator repository would require:

1. Hard coded inputs:
   • Web pipeline build properties
   • Web pipeline deployment context
2. Merge-time actions:
   1. Perform the web pipeline CI/CD actions

Approach 2

Select PR environments (either based on files changes or a label/tag/prefix on the PR) get deployed to ephemeral environments

This approach would require managing short-lived, dynamically configured environments, managed entirely within the web validator repository. Doing so would require:

1. Templatized inputs:
   • tfvars for new deployment target
   • Web pipeline build properties
   • Web pipeline deployment context
2. Template context sources:
   • Per-PR unique identifier, from within a pull_request event
   • Per-PR unique identifier, from within a push event
3. PR-time actions:
   1. Render tfvars, build properties, and deployment context
   2. Create terraform state bucket
   3. Consume the gtfs-infrastructure repository and perform the terraform build
   4. Perform the web pipeline CI/CD action
   5. Publish the resulting application URL
4. Merge-time (or PR closed time) actions
   1. Render tfvars
   2. Teardown PR deployment target
   3. Teardown terraform state bucket

Evaluation of approaches

Approach 1

Pros:

• Solutiuon implementation and environment management is simpler
• The state of main is always reflected in an environment
• GCP costs & utilization are stable

Cons:

• Merges may "stomp" on each other; that is, when two merges happen quickly in succession the state of the second merge may be deployed prior to the state of the first merge being completely validated
• Feedback from staging deployment would not be incorporated into PR reviews

Estimate: 16 hours

Approach 2

Pros:

• "Stomping" is not a problem
• Validation/feedback happens earlier in a development cycle (prior to a merge)

Cons:

• Complexity of implementation and management is substantially higher
• PRs may go out-of-sync with main, with the resulting validations being applied to a state which will never exist
• GCP costs and utilization are variable; management of bounds becomes a requirement

Estimate: 32 hours

[isabelle-dr]

@themightychris can we close this?