S3: intermittent TLS 1.3 handshake failures

awnumar commented 1 month ago

Acknowledgements

[X] I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
[X] I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

We are seeing intermittent failures establishing a TLS connection with S3 when using the v2 sdk and setting the minimum TLS version to v1.3. This is not reproducible when setting the minimum version to TLS 1.2.

We have followed the documentation available here to construct our client.

Expected Behavior

We expect the connection to reliably succeed.

Current Behavior

We occasionally see

error operation error S3: PutObject, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Put "https://bucket-name.s3.eu-west-1.amazonaws.com/test.txt?x-id=PutObject": EOF

From dumping the connection trace into Wireshare with tcpdump, we see that AWS responds to our ClientHello with a TLS Alert 21 (Close Notify) warning before it closes the connection.

Reproduction Steps

package main

import (
    "bytes"
    "context"
    "crypto/rand"
    "crypto/tls"
    "fmt"
    "net/http"
    "net/url"
    "time"

    awshttp "github.com/aws/aws-sdk-go-v2/aws/transport/http"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
    for {
        if err := do(); err != nil {
            fmt.Println("error", err)
        } else {
            fmt.Println("success")
        }
        time.Sleep(1 * time.Second)
    }
}

func do() error {
    ctx := context.Background()

    client := awshttp.NewBuildableClient().
        WithTimeout(30 * time.Second).
        WithTransportOptions(func(tr *http.Transport) {
            if tr.TLSClientConfig == nil {
                tr.TLSClientConfig = &tls.Config{}
            }
            tr.TLSClientConfig.MinVersion = tls.VersionTLS13
        })

    cfg, err := config.LoadDefaultConfig(ctx, config.WithRegion("eu-west-1"), config.WithHTTPClient(client))
    if err != nil {
        return err
    }

    c := s3.NewFromConfig(cfg)
    bucket := "bucket-name"
    key := "test.txt"

    data := make([]byte, 10000)
    if _, err := rand.Read(data); err != nil {
        return err
    }

    if _, err := c.PutObject(ctx, &s3.PutObjectInput{
        Bucket: &bucket,
        Key:    &key,
        Body:   bytes.NewReader(data),
    }); err != nil {
        return err
    }
    fmt.Println(url.Parse(fmt.Sprintf("s3://%s/%s", bucket, key)))
    return nil
}

Possible Solution

No response

Additional Information/Context

We're not able to reproduce this using the AWS CLI.

#!/bin/bash

while true
do 
    aws s3 cp ./test.txt s3://bucket-name/test.txt
    sleep 1
done

AWS Go SDK V2 Module Versions Used

The SDK versions we're using are the current latest:

go 1.22.1

require (
    github.com/aws/aws-sdk-go-v2 v1.26.1
    github.com/aws/aws-sdk-go-v2/config v1.27.11
    github.com/aws/aws-sdk-go-v2/service/s3 v1.53.1
)

require (
    github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
    github.com/aws/aws-sdk-go-v2/credentials v1.17.11 // indirect
    github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.1 // indirect
    github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.5 // indirect
    github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.5 // indirect
    github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
    github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.5 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.2 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.7 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.7 // indirect
    github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.5 // indirect
    github.com/aws/aws-sdk-go-v2/service/sso v1.20.5 // indirect
    github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.4 // indirect
    github.com/aws/aws-sdk-go-v2/service/sts v1.28.6 // indirect
    github.com/aws/smithy-go v1.20.2 // indirect
)

Compiler and Version used

go version go1.22.1 darwin/arm64

Operating System and version

Reproduced on Darwin Sonoma 14.4.1 (23E224) Arm64, and Linux/Amd64 running Amazon Linux 2023 6.1.79-99.164.amzn2023.x86_64

awnumar commented 1 month ago

I've been able to reproduce this using Curl, so I don't believe the issue is with this SDK.

curl -v "https://bucket-name.s3.eu-west-1.amazonaws.com" --tlsv1.3 --tls-max 1.3

Most of the time this works but occasionally it responds with curl: (35) OpenSSL SSL_connect: SSL_ERROR_ZERO_RETURN

So it seems like some of AWS's edge servers in this region do not support TLS 1.3, and so occasionally when we're routed to a particular server that does not support it, it will immediately close the connection.

github-actions[bot] commented 1 month ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

aws / aws-sdk-go-v2