Purchase reserved instances for OpenSearch cluster and GitLab

[hannes-ucsc]

• [ ] dev
  • [x] opensearch 4+2 nodes
  • [ ] gitlab
• [ ] anvildev
  • [x] opensearch 2+2 nodes
  • [ ] gitlab
• [ ] anvilprod
  • [x] opensearch 4+2 nodes
  • [ ] gitlab
• [ ] prod
  • [x] opensearch 4 nodes
  • [ ] gitlab

• [x] Security design review completed; the Resolution of this issue does not …
  • [x] … affect authentication; for example:
  • OAuth 2.0 with the application (API or Swagger UI)
  • Authentication of developers with Google Cloud APIs
  • Authentication of developers with AWS APIs
  • Authentication with a GitLab instance in the system
  • Password and 2FA authentication with GitHub
  • API access token authentication with GitHub
  • Authentication with
  • [x] … affect the permissions of internal users like access to
  • Cloud resources on AWS and GCP
  • GitLab repositories, projects and groups, administration
  • an EC2 instance via SSH
  • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • [x] … affect the permissions of external users like access to
  • TDR snapshots
  • [x] … affect permissions of service or bot accounts
  • Cloud resources on AWS and GCP
  • [x] … affect audit logging in the system, like
  • adding, removing or changing a log message that represents an auditable event
  • changing the routing of log messages through the system
  • [x] … affect monitoring of the system
  • [x] … introduce a new software dependency like
  • Python packages on PYPI
  • Command-line utilities
  • Docker images
  • Terraform providers
  • [x] … add an interface that exposes sensitive or confidential data at the security boundary
  • [x] … affect the encryption of data at rest
  • [x] … require persistence of sensitive or confidential data that might require encryption at rest
  • [x] … require unencrypted transmission of data within the security boundary
  • [x] … affect the network security layer; for example by
  • modifying, adding or removing firewall rules
  • modifying, adding or removing security groups
  • changing or adding a port a service, proxy or load balancer listens on
• [x] Documentation on any unchecked boxes is provided in comments below

[hannes-ucsc]

Currently getting quota error when trying to purchase reserved instances for OpeSearch in prod. Filed issue with AWS support.

[hannes-ucsc]

Quota error was resolved with the help of AWS support.

[hannes-ucsc]

I did purchase reserved instances for ElasticSearch in all AWS accounts which should reduce our ElasticSearch cost by 31% going forward e.g., in anvilprod, from $1628.00 to $1,123 per month. The term is one year with no upfront.

I decided not to purchase any reserved capacity that would cover the GitLab instance and/or Lambda. We plan to switch the GitLab EC2 instance to an instance type with an ARM architecture (Graviton) so committing to a specific instance type or type family would prevent us from doing so. Graviton instances have up to 20% more compute power for the same price. We should switch to Graviton first, and then make the commitment.

AWS also offers more flexible savings plans that don't require a commitment to a particular instance type/family but I think they are thwarted by the bursty nature of our Lambda usage (see diagram for anvilprod below). Based on Amazon's recommendations, the monthly savings in prod would be $23.91, which I don't think is worth it. These savings plans promote consistent usage, not bursts. While the GitLab instance constitutes consistent usage, our bursty reindexes with Lambda do not.

[hannes-ucsc]

I've done the low hanging fruit, the next steps are to switch GitLab to ARM (#5825) and then purchase a family-based EC2 instance savings plan.