ministryofjustice / cloud-platform

Documentation on the MoJ cloud platform
MIT License
87 stars 45 forks source link

RDS Modules Bumping - Large Storage Size DB Approach #6370

Open timckt opened 4 weeks ago

timckt commented 4 weeks ago

TL;DR:

When bumping module, users will need to take action and set the db_iops value to be at least 12000 for below 14 + 8 = 22 DBs

DBInstanceIdentifier                     AllocatedStorage   Engine       StorageType  Environment     Namespace                                     SlackChannel              InfraSupport
cloud-platform-22ccc42afd500eb2          830                postgres     gp2          production      hmpps-book-secure-move-api-production         N/A                       *
cloud-platform-2d4c157062b20f5d          10000              postgres     gp2          dev             hmpps-activities-management-dev               N/A                       *
cloud-platform-35c188a93c8a060d          500                mariadb      gp2          development     intranet-dev                                  N/A                       *
cloud-platform-43f547bdfa94ce12          550                postgres     gp2          stage           hmpps-delius-alfresco-stage                   ask-probation-webops      *
cloud-platform-59b5cf9e5de6b794          600                postgres     gp2          development     hmpps-dpr-fake-dps-service                    ask_dpr                   *
cloud-platform-5a6f44384e7e06e6          1491               postgres     gp2          preprod         hmpps-activities-management-preprod           N/A                       *
cloud-platform-5b6e8eb8ba215b7b          1500               postgres     gp2          prod            hmpps-manage-adjudications-api-prod           N/A                       *
cloud-platform-5c6377808488916a          512                postgres     gp2          preprod         offender-case-notes-preprod                   public_move-and-improve   *
cloud-platform-6c035586d92ac925          2000               postgres     gp2          preprod         hmpps-manage-adjudications-api-preprod        N/A                       *
cloud-platform-7520cc9334a6f330          16000              postgres     gp2          prod            visit-someone-in-prison-backend-svc-prod      N/A                       *
cloud-platform-78407cd5fbd86ed5          750                postgres     gp2          production      hmpps-workload-prod                           N/A                       *
cloud-platform-a066adb9989d7f7c          1500               postgres     gp2          preprod         hmpps-delius-alfresco-preprod                 ask-probation-hosting     *
cloud-platform-b6244d8c893e1d12          512                postgres     gp2          prod            offender-case-notes-prod                      public_move-and-improve   *
cloud-platform-c6e5432ed45afd19          16000              postgres     gp2          prod            visit-someone-in-prison-backend-svc-prod      N/A                       *

# Below DBs shall be in migration branch.
DBInstanceIdentifier                     AllocatedStorage   Engine       StorageType  Environment     Namespace                                     SlackChannel              InfraSupport
cloud-platform-0ba062271a739a44          300                oracle-se2   gp2          production      laa-crown-court-remuneration-production       laa-crime-higher-billing  *
cloud-platform-22fea73785debcdc          300                oracle-se2   gp2          development     laa-crown-court-litigator-fees-dev            laa-crime-higher-billing  *
cloud-platform-752ad03b27f0d355          300                oracle-se2   gp2          staging         laa-crown-court-litigator-fees-staging        laa-crime-higher-billing  *
cloud-platform-790bee95824fd5d5          300                oracle-se2   gp2          production      laa-crown-court-litigator-fees-production     laa-crime-higher-billing  *
cloud-platform-8eccc874fa0f1708          300                oracle-se2   gp2          staging         laa-crown-court-remuneration-staging          laa-crime-higher-billing  *
cloud-platform-9934ff6a725528ca          300                oracle-se2   gp2          uat             laa-crown-court-remuneration-uat              laa-crime-higher-billing  *
cloud-platform-a6e1cf58f91e81c0          300                oracle-se2   gp2          uat             laa-crown-court-litigator-fees-uat            laa-crime-higher-billing  *
cloud-platform-bb83da2ed8245f4d          300                oracle-se2   gp2          development     laa-crown-court-remuneration-dev              laa-crime-higher-billing  *

Background

We have cut the major 8.0.0 release for RDS module and we are going to bump the modules across our platform.

For all RDS DB engines except RDS for SQL Server, the baseline storage performance for gp3 volumes increases when the storage size reaches certain thresholds. This means that when user upgrade the module and switch to gp3, they may need to adjust the db_iops value accordingly to ensure optimal performance, especially for RDS instances with large volumes.

gp3 Storage Type

Database Engine Storage Size (GiB) Baseline Storage Performance Range of Provisioned IOPS Range of Provisioned Storage Throughput
Db2, MariaDB, MySQL, PostgreSQL 20–399 3,000 IOPS / 125 MiB/s N/A N/A
Db2, MariaDB, MySQL, PostgreSQL 400–65,536 12,000 IOPS / 500 MiB/s 12,000–64,000 IOPS 500–4,000 MiB/s
Oracle 20–199 3,000 IOPS / 125 MiB/s N/A N/A
Oracle 200–65,536 12,000 IOPS / 500 MiB/s 12,000–64,000 IOPS 500–4,000 MiB/s
SQL Server 20–16,384 3,000 IOPS / 125 MiB/s 3,000–16,000 IOPS 125–1,000 MiB/s

When bumping module, users will need to take action and set the db_iops value to be at least 12000 for :

The complete output are in below comment.

Proposed user journey

Approach

Which part of the user docs does this impact

Communicate changes

Questions / Assumptions

Definition of done

Reference

How to write good user stories

timckt commented 4 weeks ago
printf "%-40s %-18s %-12s %-12s %-15s %-45s %-25s %-25s\n" \
"DBInstanceIdentifier" "AllocatedStorage" "Engine" "StorageType" "Environment" "Namespace" "SlackChannel" "InfraSupport"

aws rds describe-db-instances \
    --query 'DBInstances[?AllocatedStorage > `399` && (Engine == `postgres` || Engine == `mysql` || Engine == `mariadb`) && StorageType == `gp2`]' \
    --output json | \
jq -r '.[] | [.DBInstanceIdentifier, .AllocatedStorage, .Engine, .StorageType, .DBInstanceArn] | @tsv' | \
while IFS=$'\t' read -r db_id storage engine storage_type arn; do
    tags=$(aws rds list-tags-for-resource --resource-name "$arn" --output json)
    namespace=$(echo "$tags" | jq -r '.TagList[] | select(.Key=="namespace") | .Value')
    slack_channel=$(echo "$tags" | jq -r '.TagList[] | select(.Key=="slack-channel") | .Value')
    infra_support=$(echo "$tags" | jq -r '.TagList[] | select(.Key=="infrastructure-support") | .Value')
    environment=$(echo "$tags" | jq -r '.TagList[] | select(.Key=="environment-name") | .Value')
    namespace=${namespace:-"N/A"}
    slack_channel=${slack_channel:-"N/A"}
    infra_support=${infra_support:-"N/A"}
    storage_type=${storage_type:-"N/A"}
    environment=${environment:-"N/A"}
    printf "%-40s %-18s %-12s %-12s %-15s %-45s %-25s %-25s\n" \
    "$db_id" "$storage" "$engine" "$storage_type" "$environment" "$namespace" "$slack_channel" "$infra_support"
done

The below output is as of Oct 24 20:00, removed the email address in InfraSupport here for data privacy. We have 14 DB as of Oct 24 20:00 in this case.


DBInstanceIdentifier                     AllocatedStorage   Engine       StorageType  Environment     Namespace                                     SlackChannel              InfraSupport
cloud-platform-22ccc42afd500eb2          830                postgres     gp2          production      hmpps-book-secure-move-api-production         N/A                       *
cloud-platform-2d4c157062b20f5d          10000              postgres     gp2          dev             hmpps-activities-management-dev               N/A                       *
cloud-platform-35c188a93c8a060d          500                mariadb      gp2          development     intranet-dev                                  N/A                       *
cloud-platform-43f547bdfa94ce12          550                postgres     gp2          stage           hmpps-delius-alfresco-stage                   ask-probation-webops      *
cloud-platform-59b5cf9e5de6b794          600                postgres     gp2          development     hmpps-dpr-fake-dps-service                    ask_dpr                   *
cloud-platform-5a6f44384e7e06e6          1491               postgres     gp2          preprod         hmpps-activities-management-preprod           N/A                       *
cloud-platform-5b6e8eb8ba215b7b          1500               postgres     gp2          prod            hmpps-manage-adjudications-api-prod           N/A                       *
cloud-platform-5c6377808488916a          512                postgres     gp2          preprod         offender-case-notes-preprod                   public_move-and-improve   *
cloud-platform-6c035586d92ac925          2000               postgres     gp2          preprod         hmpps-manage-adjudications-api-preprod        N/A                       *
cloud-platform-7520cc9334a6f330          16000              postgres     gp2          prod            visit-someone-in-prison-backend-svc-prod      N/A                       *
cloud-platform-78407cd5fbd86ed5          750                postgres     gp2          production      hmpps-workload-prod                           N/A                       *
cloud-platform-a066adb9989d7f7c          1500               postgres     gp2          preprod         hmpps-delius-alfresco-preprod                 ask-probation-hosting     *
cloud-platform-b6244d8c893e1d12          512                postgres     gp2          prod            offender-case-notes-prod                      public_move-and-improve   *
cloud-platform-c6e5432ed45afd19          16000              postgres     gp2          prod            visit-someone-in-prison-backend-svc-prod      N/A                       *```
timckt commented 4 weeks ago

aws rds describe-db-instances \ --query 'DBInstances[?AllocatedStorage > 199 && (Engine == oracle-se2) && StorageType == gp2]' \ --output json | \ jq -r '.[] | [.DBInstanceIdentifier, .AllocatedStorage, .Engine, .StorageType, .DBInstanceArn] | @tsv' | \ while IFS=$'\t' read -r db_id storage engine storage_type arn; do tags=$(aws rds list-tags-for-resource --resource-name "$arn" --output json) namespace=$(echo "$tags" | jq -r '.TagList[] | select(.Key=="namespace") | .Value') slack_channel=$(echo "$tags" | jq -r '.TagList[] | select(.Key=="slack-channel") | .Value') infra_support=$(echo "$tags" | jq -r '.TagList[] | select(.Key=="infrastructure-support") | .Value') environment=$(echo "$tags" | jq -r '.TagList[] | select(.Key=="environment-name") | .Value') namespace=${namespace:-"N/A"} slack_channel=${slack_channel:-"N/A"} infra_support=${infra_support:-"N/A"} storage_type=${storage_type:-"N/A"} environment=${environment:-"N/A"} printf "%-40s %-18s %-12s %-12s %-15s %-45s %-25s %-25s\n" \ "$db_id" "$storage" "$engine" "$storage_type" "$environment" "$namespace" "$slack_channel" "$infra_support" done


The below output is as of Oct 24 20:00, removed the email address in `InfraSupport` here for data privacy
We have **8** DB as of Oct 24 20:00 in this case.
Those DB shall be in `migration` branch. 

DBInstanceIdentifier AllocatedStorage Engine StorageType Environment Namespace SlackChannel InfraSupport cloud-platform-0ba062271a739a44 300 oracle-se2 gp2 production laa-crown-court-remuneration-production laa-crime-higher-billing cloud-platform-22fea73785debcdc 300 oracle-se2 gp2 development laa-crown-court-litigator-fees-dev laa-crime-higher-billing cloud-platform-752ad03b27f0d355 300 oracle-se2 gp2 staging laa-crown-court-litigator-fees-staging laa-crime-higher-billing cloud-platform-790bee95824fd5d5 300 oracle-se2 gp2 production laa-crown-court-litigator-fees-production laa-crime-higher-billing cloud-platform-8eccc874fa0f1708 300 oracle-se2 gp2 staging laa-crown-court-remuneration-staging laa-crime-higher-billing cloud-platform-9934ff6a725528ca 300 oracle-se2 gp2 uat laa-crown-court-remuneration-uat laa-crime-higher-billing cloud-platform-a6e1cf58f91e81c0 300 oracle-se2 gp2 uat laa-crown-court-litigator-fees-uat laa-crime-higher-billing cloud-platform-bb83da2ed8245f4d 300 oracle-se2 gp2 development laa-crown-court-remuneration-dev laa-crime-higher-billing ```

sj-williams commented 1 week ago
DBInstanceIdentifier                     AllocatedStorage   Engine       StorageType  Environment     Namespace
cloud-platform-22ccc42afd500eb2          913                postgres     gp2          production      hmpps-book-secure-move-api-production
cloud-platform-2d4c157062b20f5d          10000              postgres     gp2          dev             hmpps-activities-management-dev
cloud-platform-43f547bdfa94ce12          550                postgres     gp2          stage           hmpps-delius-alfresco-stage
cloud-platform-59b5cf9e5de6b794          600                postgres     gp2          development     hmpps-dpr-fake-dps-service
cloud-platform-5a6f44384e7e06e6          1491               postgres     gp2          preprod         hmpps-activities-management-preprod
cloud-platform-5b6e8eb8ba215b7b          1500               postgres     gp2          prod            hmpps-manage-adjudications-api-prod
cloud-platform-6c035586d92ac925          2000               postgres     gp2          preprod         hmpps-manage-adjudications-api-preprod
cloud-platform-7520cc9334a6f330          16000              postgres     gp2          prod            visit-someone-in-prison-backend-svc-prod
cloud-platform-c6e5432ed45afd19          16000              postgres     gp2          prod            visit-someone-in-prison-backend-svc-prod
sj-williams commented 3 days ago

Down to 7:

DBInstanceIdentifier
cloud-platform-22ccc42afd500eb2
cloud-platform-43f547bdfa94ce12
cloud-platform-59b5cf9e5de6b794
cloud-platform-5b6e8eb8ba215b7b
cloud-platform-6c035586d92ac925
cloud-platform-7520cc9334a6f330
cloud-platform-c6e5432ed45afd19