Need to analyze and handle new SSL certs in RDS (Deadline August)

[mlissner]

AWS sent a lengthy email with the following details:

Hello,

[AWS Health may periodically trigger reminder notifications about this communication.]

You are receiving this message because your AWS Account has one or more Amazon RDS, or Amazon Aurora database instances in the US-WEST-2 Region using a SSL/TLS Certificate that is expiring on August 22, 2024.

A list of your affected resources can be found in the 'Affected resources' tab of your AWS Health Dashboard.

This is a follow-up notification for SSL/TLS CA certification expiration. If you believe you have already finished this work and still received this email, it is likely because you created new instances using the 2019 Certificate Authority (CA). After January 25, 2024 all newly created instances that do not explicitly specify a different CA will use the ‘rds-ca-rsa2048-g1’ CA. For information on setting an account level CA override, see the modify-certificates API documentation [1].

If your applications connect to these instances using the SSL/TLS protocol, you will need to take action before August 22, 2024 to prevent connectivity failures to your existing database instances. Even if you do not currently use SSL for your connections, you could still be affected if your databases server certificate expires, so we still recommend updating your CA.

To protect your communications with your database instances, a CA generates time-bound certificates that are checked by your database client software to authenticate any database instance before exchanging information. Following industry best practices, AWS renews the CA and creates new certificates on a routine basis to ensure customer connections are properly protected for years to come. The current CA in the US-WEST-2 Region will expire on August 22, 2024. Before this date you will need to update your DB server certificate. The following is the general process to do this:

First, update your application clients with the new certificate, if your application client is using a trust store then add the new CA certificates into the trust stores of your client applications. RDS provides download links to the CA certificates in our User Guide [2]. For more detailed instructions on updating the trust stores on your client application see our documentation [3].

Second, update the certificate on all your affected database instances to one of the newly issued CAs. ‘rds-ca-rsa2048-g1’ is the default recommended CA because there is no algorithm change. The other CAs use new key algorithms so it could require more testing of your client setup to ensure compatibility. For more information on the new CAs see our documentation [4].

Third, if you want to use a different CA than the default ‘rds-ca-rsa2048-g1’, you will need to set an account level CA override so your new instances will use the CA of your choice. To do this a modify-certificates API is available that will allow you to override the default CA on newly created database instances to either the old or one of the new CAs. This override will only apply while the CA you are overriding to is valid. To use this API you will need to be running the AWS CLI version 1.17 or later. For more information see the modify-certificates API documentation [1]. There is also a describe-certificates API [5], that will indicate your current default CA override if you have one set. To set a specific CA during instance creation use the ca-certificate-identifier option on the create-db-instance API to create a DB instance with a specific CA. For more information, see the create-db-instance API documentation [6].

For more detailed instructions on how to perform these updates, please see the Amazon RDS instances [7] and Amazon Aurora instances [8] documentation.

Please be aware of the following timeline:

1. As soon as possible - You should update your client trust store, then you can update your instances server certificate to one generated by one of the new CAs.

2. After January 26, 2024 - any new database instances default to using the ‘rds-ca-rsa2048-g1’ certificate. If you wish for new instances to use a different certificate, you can specify which certificate to use with the AWS console or the AWS CLI. For more information, see the create-db-instance API documentation [6].

3. After August 22, 2024 - ‘rds-ca-2019’ will expire. You will need to take action before August 22, 2024, to prevent connectivity failures to your existing database instances.

For more information see our blog [9], and if you have questions or concerns, please contact AWS Support [10].

[1] https://docs.aws.amazon.com/cli/latest/reference/rds/modify-certificates.html [2] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html#UsingWithRDS.SSL.CertificatesAllRegions [3] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html#UsingWithRDS.SSL-certificate-rotation-updating [4] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html#UsingWithRDS.SSL.RegionCertificateAuthorities [5] https://docs.aws.amazon.com/cli/latest/reference/rds/describe-certificates.html [6] https://docs.aws.amazon.com/cli/latest/reference/rds/create-db-instance.html [7] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html [8] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/UsingWithRDS.SSL-certificate-rotation.html [9] https://aws.amazon.com/blogs/aws/rotate-your-ssl-tls-certificates-now-amazon-rds-and-amazon-aurora-expire-in-2024/ [10] https://console.aws.amazon.com/support/home

Sincerely, Amazon Web Services

Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210

---

Reference: https://phd.aws.amazon.com/phd/home?region=us-east-1#/event-log?eventID=arn:aws:health:us-west-2::event/RDS/AWS_RDS_PLANNED_LIFECYCLE_EVENT/AWS_RDS_PLANNED_LIFECYCLE_EVENT_7894558cf84495ccf0a336cb0190015d010a31d5e0a6556ba84dd70a880c02bf&eventTab=details

I asked Alberto if he had thoughts about this, and he replied:

Yeah, well it depends on whether we need to take action or not.

They say we have RDS or Aurora instances in US-WEST-2 where there is a new CA rds-ca-rsa2048-g1 while the old one rds-ca-2019 will expire on August 22, 2024.

Based on this, we need to ensure the new CA is included in the trust store of our applications and any clients that connect to the affected databases.

For instance, our Django connection currently uses "require": "OPTIONS": { "sslmode": env("DB_SSL_MODE", default="require"), }

This configuration means the database should support SSL, but we're not using the "sslrootcert" option, which should point to the current CA, rds-ca-2019. This means that rds-ca-2019 is included in a lower-level trust store, like the one on Linux, possibly included during the EC2 instance creation.

So, it appears we need to ensure our instances are prepared to trust the new CA before August 22, 2024.

We can create a test RDS instance with its CA set to the new rds-ca-rsa2048-g1, as described in this article. We could use this instance to test the connection in a non-production environment, such as a maintenance pod. This way, we can determine if further actions are needed. If we can connect to the RDS instance using the new CA, it means rds-ca-rsa2048-g1 is already included in the instance's trust store.

If we can't connect to the RDS instance, we'll need to use the new CA for our connections. We'd then need to mount the CA as a volume so it's accessible within pods, allowing us to add it to the sslrootcert setting.

After confirming that we can perform connections using the new CA, we should update our current RDS instances to the new CA before the expiration date so we can handle any not considered issues before it expires and it is not possible to go back to rds-ca-2019.

Additionally, if any customer replicas are also hosted on RDS, we should ensure they add the new CA to their applications to maintain connectivity before the old one expires. Let me know if you want me to investigate anything else here.

Many of our new RDS instances use the new certs, so we can use those to test. This issue us to make sure everything is/will be OK.

[blancoramiro]

Just leaving an update here.

Instances pending to update:

• bots-law
• cl-replica
• prod

[blancoramiro]

Hey @mlissner. All databases are now using CA rds-ca-rsa2048-g1 (latest and default).

Will keep an eye open for any issues that might be related to this.

Ty! Closing this issue.

[mlissner]

Can you please say a few words about how you fixed it, concerns you had, and whether there any issues?

[blancoramiro]

Sure. Since Courtlistener and bots-law applications have the SSL_MODE variable set to require it means that TLS is used but the server certificated is not verified against a CA.

Weather to check or not the server certificate is configured client side.

Main concerns were related to affecting the applications during the update of the certificate however no interruptions are noticeable during the change, current sessions are not closed and newer ones can connect using the new certificate.

I tested this with lower importance instances before moving into more critical ones.

More info on postgresql ssl configurations here

More information on the different ssl modes and their security implications here