Adjust TLS access rules for s3 buckets to reject TLS 1.0 and 1.1

[eddierubeiz]

This summer Amazon will be updating the TLS configuration for all AWS service API endpoints to a minimum of version TLS 1.2. This update means we and our users will no longer be able to use TLS versions 1.0 and 1.1 when accessing S3.

https://aws.amazon.com/blogs/security/tls-1-2-required-for-aws-endpoints/

Amazon sent us an action required email on 4/6/23 requesting us to add a clause to our S3 access configuration rules to require TLS 1.2 for all access to scihist-digicoll-production-derivatives.

I assume we would want to make the change to all our s3 buckets if we're doing it at all.

We're pretty sure we're been using up-to-date TLS protocols in all our dealings with S3, but we should verify that.

[eddierubeiz]

My interpretation of the email is "lock down the S3 buckets soon, and find out if anything breaks. If you don't do it now, we will lock them down for you in July -- whether you like it or not".

[jrochkind]

Where'd you find the July deadline? If you have a specific date in July, can you add to this ticket for reference?

It looks like Office 365 for instance already rejects TLS prior to 1.2.

So I'm not worried about any of our users who have browsers which connect to the bucket to load assets.

It is slightly more possible that some internal service running in ruby (or other?) is still running an old TLS. It probably would be most responsible of us to test it out before July, so if we find a problem we can reverse it and fix it with less urgency.

More responsible, but also not super urgent, at worst it will break our ingest or fixity checks or something until we fix it. It will just save us from the possibility of having to drop everything to fix something in July. :)

[eddierubeiz]

The date's mentioned at the link above.

It's not exactly a deadline - the exact language is:

To minimize the availability impact of requiring TLS 1.2, AWS is rolling out the changes on an endpoint-by-endpoint basis over the next year, starting now and ending in June 2023. Before making these potentially breaking changes, we monitor for connections that are still using TLS 1.0 or TLS 1.1. If you are one of the AWS customers who may be impacted, we will notify you on your AWS Health Dashboard, and by email. After June 28, 2023, AWS will update our API endpoint configuration to remove TLS 1.0 and TLS 1.1, even if you still have connections using these versions.

See also: https://health.aws.amazon.com/health/home#/account/dashboard/other-notifications

My assumption is if anyone requests anything from any S3 bucket using the outdated protocol, the owner of the bucket gets the email we got.

[jrochkind]

These are all from external public users (probably bots) on our public S3 buckets.

AWS is eventually going to reject these old TLS versions -- but in the meantime, it periodically warns us about them, with annoying emails.

Let's actually configure (via terraform) to start rejecting these now, so AWS will stop sending us the warning emails.

As a best practice, and to prepare for our enforcement of TLS 1.2 or higher, we recommend you proactively enforce a minimum of TLS 1.2 directly on all of your shared S3 bucket(s). You may do this by applying a bucket policy with the s3:TlsVersion condition key as per the documented this Knowledge Center article https://aws.amazon.com/premiumsupport/knowledge-center/s3-enforce-modern-tls/

Hello,

We have identified TLS 1.0 or TLS 1.1 connections to Amazon Simple Storage Service (Amazon S3) objects hosted in your account, which must be immediately updated for these connections to maintain their access to your S3 objects. Please update your client software as soon as possible to use TLS 1.2 or higher to avoid an availability impact. We recommend considering the time needed to verify your changes in a staging environment before introducing them into production.

As of June 28, 2023, we have begun deploying updates to the TLS configuration for all AWS API endpoints to a minimum of version TLS 1.2 even if you still have connections using these versions. These deployments will complete by no later than December 31, 2023. This update removes the ability to use TLS versions 1.0 and 1.1 with all AWS APIs in all AWS Regions [1]. 

What actions can I take to maintain access?
To avoid potential interruption, you must update all client software accessing your Amazon S3 objects using TLS 1.0 or 1.1, to use TLS 1.2 or higher. If you are unable or would prefer to not update all impacted clients, we recommend replacing direct client access to the S3 objects with use of a proxy, such as an Amazon CloudFront distribution. This will allow clients to access your S3 objects via Amazon CloudFront using any TLS version you choose to allow. Amazon CloudFront will forward the calls to your S3 objects using TLS 1.2 or higher. For more guidance for how to setup your CloudFront distribution to front your S3 object access, please review this Knowledge Center article [2].

How can I determine the client(s) I need to update?
We have provided the affected S3 bucket(s) in your account following this messaging. In order to gather additional information about the affected objects and user agents performing these calls, we recommend enabling Amazon CloudTrail data events on the affected S3 bucket(s) [3] [4]. The information contained in the S3 data events will help you pinpoint your client software that is responsible for using TLS 1.0 or TLS 1.1, so you may update it accordingly. Additionally, our related AWS Security blog post [1] provides information on how you may use TLS information in the CloudTrail tlsDetails field. Please note there is an associated cost for enabling CloudTrail data events, please see the CloudTrail pricing page for more detail [5]. Another alternative is to use Amazon S3 server-access logs, see the S3 Logging options page for more details and pricing information [6].

How can I enforce connections to my bucket(s) be over TLSv1.2 and above?
As a best practice, and to prepare for our enforcement of TLS 1.2 or higher, we recommend you proactively enforce a minimum of TLS 1.2 directly on all of your shared S3 bucket(s). You may do this by applying a bucket policy with the s3:TlsVersion condition key as per the documented this Knowledge Center article [7]

If you need further guidance or assistance, please contact AWS Support [8] or your Technical Account Manager.

[1] https://aws.amazon.com/blogs/security/tls-1-2-required-for-aws-endpoints
[2] https://aws.amazon.com/premiumsupport/knowledge-center/s3-access-old-tls/
[3] https://docs.aws.amazon.com/AmazonS3/latest/userguide/cloudtrail-logging-s3-info.html#cloudtrail-object-level-tracking
[4] https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-cloudtrail-logging-for-s3.html#enable-cloudtrail-events
[5] https://aws.amazon.com/cloudtrail/pricing/
[6] https://docs.aws.amazon.com/AmazonS3/latest/userguide/logging-with-S3.html
[7] https://aws.amazon.com/premiumsupport/knowledge-center/s3-enforce-modern-tls/
[8] https://aws.amazon.com/support

Please see the following for S3 buckets in which object-level calls were made over TLS 1.0 or TLS 1.1 connections between July 17, 2023 and July 25, 2023 (the UserAgent may be truncated due to a limit in the number of characters that can be displayed):  

Connections details will be in the following format:
Region | Bucket name(s) | APIAction | TLSVersion | NumCalls | UserAgent
us-east-1 | scihist-digicoll-production-derivatives | REST.GET.OBJECT | TLSv1 | 1 | [Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Win64; x64; Trident/4.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.

Sincerely,
Amazon Web Services

Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210

---
Reference: https://phd.aws.amazon.com/phd/home?region=us-east-1#/event-log?eventID=arn:aws:health:global::event/SECURITY/AWS_SECURITY_TLS_DEPRECATION_NOTIFICATION/AWS_SECURITY_TLS_DEPRECATION_NOTIFICATION_744d40956f6fa5811c47210e27889d00e2345ed54fddd667d34e4680afe01d5f&eventTab=details

[eddierubeiz]

This is applied in staging. Still need to apply in production (next week).

[eddierubeiz]

Applied in production this morning.

[eddierubeiz]

Last night we got the following from AWS:

Please see the following for S3 buckets in which object-level calls were made over TLS 1.0 or TLS 1.1 connections between September 17, 2023 and October 1, 2023 (the UserAgent may be truncated due to a limit in the number of characters that can be displayed):

Connections details will be in the following format:
Region | Bucket name(s) | APIAction | TLSVersion | NumCalls | UserAgent
us-east-1 | scihist-digicoll-production-originals | REST.GET.OBJECT | TLSv1 | 1 | [Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36]

This doesn't square with my understanding of what's going on. I'd like to spend some time investigating

• whether or not we are rejecting TLSv1 connections (the point of this issue)
• whether the above message concerns
  • a request that our servers responded to (bad)
  • a request that was in fact rejected due to our current policy, but still somehow triggered the message above (fine).

[eddierubeiz]

Start here: https://s3.console.aws.amazon.com/s3/object/scihist-digicoll-production-originals?region=us-east-1&prefix=asset/xyz/abc.tif

From the AWS UI, select: Object actions -> Share with a presigned URL.

presigned_url="https://scihist-digicoll-production-originals.s3.us-east-1.amazonaws.com/blablabla"

curl --verbose $presigned_url  --output file.tiff
    (downloads file)

curl --verbose   --tlsv1.1 --tls-max 1.1 $presigned_url --output file_2.tiff
    HTTP/1.1 403 Forbidden

[eddierubeiz]

So we are indeed denying TLSv1 connections with 403.

On rereading the email, it does say AWS has "identified TLS 1.0 or TLS 1.1 connections to Amazon Simple Storage Service (Amazon S3) objects hosted in [our] account". The message does not specify whether the connections were successful.

It seems a mere GET request was enough to trigger the warning email, even if that request was in fact denied with a 403.

[jrochkind]

This is annoying if we're doing the thing they told us to do, and they're going to keep sending us regular emails that we are supposed to ignore? This is hard for me to believe, I still keep thinking there must be something we're not rejecting that we could be!

Do you want to contact AWS support and ask them if there's any way to turn off these warnings, or any configuration we can do to turn them off? We obviously can't control what requests people who aren't us make to our public buckets!

[eddierubeiz]

I like this idea, yeah... if only to categorically rule out the possibility that there's anything more we could be doing.

[eddierubeiz]

Odd -- I went to https://support.console.aws.amazon.com/support/home?region=us-east-1 and tried creating a technical case. It says:

Technical support is unavailable under the Basic Support plan
Upgrade your plan to gain access to AWS technical support, architectural guidance, and more.

[eddierubeiz]

We decided at today's tech meeting that we were not going to try to file a support ticket. We're all satisfied that the only thing that is broken is AWS's misleading communications.