Update RDS database certificates by August 22, 2024

[jimleroyer]

Description

As a devops, I need to update the database certificates in all environments and products so that it remains operational and secure.

WHY are we building?

Keep our security certificates for database communication up to date.

WHAT are we building?

Updating the security certificates.

VALUE created by our solution

Security and stability.

Acceptance Criteria

• [x] consult with security team and/or SRE about using the default cert vs a custom one.
• [x] Evaluate the certificate we want, if not the default one, i.e. rds-ca-rsa2048-g1.
• [x] Switch the certificate for the notify database in dev.
• [x] Switch the certificate for the notify database in staging.
• [x] Switch the certificate for the notify database in production.
• [x] Switch the certificate for the blazer database in dev (might not be present in dev environment).
• [x] Switch the certificate for the blazer database in staging.
• [ ] Switch the certificate for the blazer database in production.

QA Steps

• [x] Notify database communication is functional in dev.
• [x] Notify database communication is functional in staging.
• [ ] Notify database communication is functional in production.
• [x] Blazer database communication is functional in dev (might not be present in dev environment).
• [x] Blazer database communication is functional in staging.

• [ ] Blazer database communication is functional in production.

  Additional information:

Pat's magic 🧙‍♂️

Command used to upgrade the certification using the default certificate. Choosing a non-default certificate would require more operations such as setting the default account certificate.

aws rds modify-db-instance \
    --db-instance-identifier $INSTANCE_NAME \
    --ca-certificate-identifier rds-ca-rsa2048-g1 \
    --apply-immediately

AWS email

This is the correspondence we received from AWS:

You are receiving this message because your AWS Account has one or more Amazon RDS, or Amazon Aurora database instances in the CA-CENTRAL-1 Region using a SSL/TLS Certificate that is expiring on August 22, 2024.

A list of your affected resources can be found in the 'Affected resources' tab of your AWS Health Dashboard.

This is a follow-up notification for SSL/TLS CA certification expiration. If you believe you have already finished this work and still received this email, it is likely because you created new instances using the 2019 Certificate Authority (CA). After January 25, 2024 all newly created instances that do not explicitly specify a different CA will use the ‘rds-ca-rsa2048-g1’ CA. For information on setting an account level CA override, see the modify-certificates API documentation [1].

If your applications connect to these instances using the SSL/TLS protocol, you will need to take action before August 22, 2024 to prevent connectivity failures to your existing database instances. Even if you do not currently use SSL for your connections, you could still be affected if your databases server certificate expires, so we still recommend updating your CA.

To protect your communications with your database instances, a CA generates time-bound certificates that are checked by your database client software to authenticate any database instance before exchanging information. Following industry best practices, AWS renews the CA and creates new certificates on a routine basis to ensure customer connections are properly protected for years to come. The current CA in the CA-CENTRAL-1 Region will expire on August 22, 2024. Before this date you will need to update your DB server certificate. The following is the general process to do this:

First, update your application clients with the new certificate, if your application client is using a trust store then add the new CA certificates into the trust stores of your client applications. RDS provides download links to the CA certificates in our User Guide [2]. For more detailed instructions on updating the trust stores on your client application see our documentation [3].

Second, update the certificate on all your affected database instances to one of the newly issued CAs. ‘rds-ca-rsa2048-g1’ is the default recommended CA because there is no algorithm change. The other CAs use new key algorithms so it could require more testing of your client setup to ensure compatibility. For more information on the new CAs see our documentation [4].

Third, if you want to use a different CA than the default ‘rds-ca-rsa2048-g1’, you will need to set an account level CA override so your new instances will use the CA of your choice. To do this a modify-certificates API is available that will allow you to override the default CA on newly created database instances to either the old or one of the new CAs. This override will only apply while the CA you are overriding to is valid. To use this API you will need to be running the AWS CLI version 1.17 or later. For more information see the modify-certificates API documentation [1]. There is also a describe-certificates API [5], that will indicate your current default CA override if you have one set. To set a specific CA during instance creation use the ca-certificate-identifier option on the create-db-instance API to create a DB instance with a specific CA. For more information, see the create-db-instance API documentation [6].

For more detailed instructions on how to perform these updates, please see the Amazon RDS instances [7] and Amazon Aurora instances [8] documentation.

Please be aware of the following timeline:

1. As soon as possible - You should update your client trust store, then you can update your instances server certificate to one generated by one of the new CAs.

2. After January 26, 2024 - any new database instances default to using the ‘rds-ca-rsa2048-g1’ certificate. If you wish for new instances to use a different certificate, you can specify which certificate to use with the AWS console or the AWS CLI. For more information, see the create-db-instance API documentation [6].

3. After August 22, 2024 - ‘rds-ca-2019’ will expire. You will need to take action before August 22, 2024, to prevent connectivity failures to your existing database instances.

For more information see our blog [9], and if you have questions or concerns, please contact AWS Support [10].

[1] https://docs.aws.amazon.com/cli/latest/reference/rds/modify-certificates.html [2] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html#UsingWithRDS.SSL.CertificatesAllRegions [3] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html#UsingWithRDS.SSL-certificate-rotation-updating [4] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html#UsingWithRDS.SSL.RegionCertificateAuthorities [5] https://docs.aws.amazon.com/cli/latest/reference/rds/describe-certificates.html [6] https://docs.aws.amazon.com/cli/latest/reference/rds/create-db-instance.html [7] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html [8] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/UsingWithRDS.SSL-certificate-rotation.html [9] https://aws.amazon.com/blogs/aws/rotate-your-ssl-tls-certificates-now-amazon-rds-and-amazon-aurora-expire-in-2024/ [10] https://console.aws.amazon.com/support/home

[sastels]

From the AWS docs:

Amazon RDS provides the following CAs to sign the DB server certificate for a DB instance.

rds-ca-2019 (currently in use, expiring soon)

Uses a certificate authority with RSA 2048 private key algorithm and SHA256 signing algorithm. This CA expires in 2024 and doesn't support automatic server certificate rotation. If you are using this CA and want to keep the same standard, we recommend that you switch to the rds-ca-rsa2048-g1 CA.

rds-ca-rsa2048-g1 (recommended replacement)

Uses a certificate authority with RSA 2048 private key algorithm and SHA256 signing algorithm in most AWS Regions.

In the AWS GovCloud (US) Regions, this CA uses a certificate authority with RSA 2048 private key algorithm and SHA384 signing algorithm.

This CA remains valid for longer than the rds-ca-2019 CA. This CA supports automatic server certificate rotation.

rds-ca-rsa4096-g1

Uses a certificate authority with RSA 4096 private key algorithm and SHA384 signing algorithm. This CA supports automatic server certificate rotation.

rds-ca-ecc384-g1

Uses a certificate authority with ECC 384 private key algorithm and SHA384 signing algorithm. This CA supports automatic server certificate rotation.

[sastels]

posted in #security-privacy for feedback.

[sastels]

Moved to Blocked while we're waiting on feedback.

[sastels]

Notes from Slack discussion:

• Standard is RSA 2048 right now, RSA 3072 by 2030.
• Cheng suggested we just move to RSA 4096 and be done with it
• decided that we'd do some perf tests with RSA 2048 (ie current rds-ca-2019) vs RSA 4096 (rds-ca-rsa4096-g1). If there's no significant performance difference we'll just upgrade to rds-ca-rsa4096-g1

[ben851]

No need to update apps or anything due to RDS proxy: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html

[ben851]

Created a PR to upgrade staging to 2048 bit

[ben851]

@sastels and I will run a perf test this morning to get a baseline in before merging the above PR.

[sastels]

We shall test in the group session today!

[P0NDER0SA]

testiung was attempted but the change didn't take. need to set an apply-immediately tag to make sure it takes. we will attempt this again today. we have a script that sends 1 email per second to soak test.

[sastels]

cert change didn't take :/ need to enable an "apply now" flag on RDS for it to work (?) - whii try again today with a soak test running. 😅

[ben851]

[ben851]

With apply immediately, the certificate took, and there was no downtime! Proceeding with upgrading the blazer db, and setting notify db to 4096 since that's working now. Will then run a performance test.

[ben851]

Perf test ran against 4096 certs, no change in performance

[ben851]

https://github.com/cds-snc/notification-terraform/pull/1224

[P0NDER0SA]

This one was done and worked on Staging. Hoping to get some time with Jimmy to get this done on Prod. The PR commented above is the one for prod.

[P0NDER0SA]

This will get merged to Prod today and needs to get QA'd

[ben851]

@Pond QA'ed

[P0NDER0SA]

It worked! just checked the cert and it's upgraded on prod.