Deploying `shared` component is slow

[hannes-ucsc]

… now that it includes the mirroring of images.

The GitLab image is huge (> 1GiB) so mirroring it to ECR will take an hour or more on a consumer internet connection (we all work from home).

• [ ] Security design review completed; the Resolution of this issue does not …
  • [ ] … affect authentication; for example:
  • OAuth 2.0 with the application (API or Swagger UI)
  • Authentication of developers with Google Cloud APIs
  • Authentication of developers with AWS APIs
  • Authentication with a GitLab instance in the system
  • Password and 2FA authentication with GitHub
  • API access token authentication with GitHub
  • Authentication with
  • [ ] … affect the permissions of internal users like access to
  • Cloud resources on AWS and GCP
  • GitLab repositories, projects and groups, administration
  • an EC2 instance via SSH
  • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • [ ] … affect the permissions of external users like access to
  • TDR snapshots
  • [ ] … affect permissions of service or bot accounts
  • Cloud resources on AWS and GCP
  • [ ] … affect audit logging in the system, like
  • adding, removing or changing a log message that represents an auditable event
  • changing the routing of log messages through the system
  • [ ] … affect monitoring of the system
  • [ ] … introduce a new software dependency like
  • Python packages on PYPI
  • Command-line utilities
  • Docker images
  • Terraform providers
  • [ ] … add an interface that exposes sensitive or confidential data at the security boundary
  • [ ] … affect the encryption of data at rest
  • [ ] … require persistence of sensitive or confidential data that might require encryption at rest
  • [ ] … require unencrypted transmission of data within the security boundary
  • [ ] … affect the network security layer; for example by
  • modifying, adding or removing firewall rules
  • modifying, adding or removing security groups
  • changing or adding a port a service, proxy or load balancer listens on
• [ ] Documentation on any unchecked boxes is provided in comments below

[hannes-ucsc]

We should do the mirroring inside AWS, obviously, but that would require an EC2 instance. We have an EC2 instance in each AWS account but it is provisioned by the gitlab component which depends on the shared component. So a classic chicken-vs-egg problem. Additionally, the GitLab instance does not have enough permission to deploy the gitlab or shared components, either, for good reason.

We could start allowing operators with slow uplinks to maintain their own bastion EC2 instance but then that instance would reside inside the authorization boundary and would have to be subject to strict compliance requirements.

There really is no good answer. My uplink was fast enough to do the initial mirroring of all images in all deployments. It required a bit of planning ahead about timing.

My recommendation is to

1) ensure that the STS token is fresh before deploying shared (#5302 will add documentation to recommend that as a general best practice for operators)

2) Have a separate Azul checkout to run operator tasks from so that the operator can continue to work on other things

Alternatively, the operator could drive to campus or WRP, hop on eduroam and perform the task from there.

[dsotirho-ucsc]

Spike to estimate monthly cost for Comcast plan upgrades & test image mirroring to ECR from Eduroam at WRP.