search

aws / aws-sdk-js-v3

Modularized AWS SDK for JavaScript.
Apache License 2.0
2.98k stars 560 forks source link

Error: Client network socket disconnected before secure TLS connection was established #5974

Closed ennioVisco closed 3 months ago

ennioVisco commented 3 months ago

Checkboxes for prior research

Note: a fascinating related issue is this one: https://github.com/aws/aws-sdk-js/issues/3591, although in our case the same error is gained both when the code is run locally and via a lambda.

Describe the bug

Hello,

It's been now a couple of weeks of debugging trying to address this issue, and we don't seem to be able to find a way around it.

We have a public s3 bucket, that we fail to access, getting the the title error, but this is a recent behaviour, it used to work perfectly, and in fact we have an old lambda executing a similar code that still works.

What could possibly be the problem?

SDK version number

@aws-sdk/client-s3: 3.550.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.9.0

Reproduction Steps

We want to simply access an object from it via the JS api, the code is more or less the following:

import {  GetObjectCommand, S3Client } from "@aws-sdk/client-s3";

async function endpoint() {
 // ...
  const client = new S3Client({
    region: "eu-west-1"
  });
  const getObjectCommand = new GetObjectCommand({
        Bucket: "BUCKET_NAME",
        Key: "FILE_KEY",
      });
  const result = await client.send(getObjectCommand);
// ...
}

SInce the bucket is public, I believe this should be reproducible by anyone.

Of course if you try to access it directly via browser it works ~LINK REMOVED~.

Observed Behavior

Running the previous code, returns the following error (with extra logging enabled)

endpoints Initial EndpointParams: {
  "ForcePathStyle": false,
  "UseArnRegion": false,
  "DisableMultiRegionAccessPoints": false,
  "Accelerate": false,
  "DisableS3ExpressSessionAuth": false,
  "UseGlobalEndpoint": false,
  "UseFIPS": false,
  "Region": "eu-west-1",
  "UseDualStack": false,
  "Bucket": "BUCKET_NAME",
  "Key": "FILE_KEY"
}
endpoints evaluateCondition: isSet($Region) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: substring($Bucket, 0, 6, true) = tadata
endpoints assign: bucketSuffix := tadata
endpoints evaluateCondition: stringEquals($bucketSuffix, --x-s3) = false
endpoints evaluateCondition: not(isSet($Bucket)) = false
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: substring($Bucket, 49, 50, true) = null
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($ForcePathStyle, false) = true
endpoints evaluateCondition: aws.isVirtualHostableS3Bucket($Bucket, false) = true
endpoints evaluateCondition: aws.partition($Region) = {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "Europe (Ireland)"
}
endpoints assign: partitionResult := {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "implicitGlobalRegion": "us-east-1",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "Europe (Ireland)"
}
endpoints evaluateCondition: isValidHostLabel($Region, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = true
endpoints evaluateCondition: stringEquals($Region, aws-global) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = true
endpoints evaluateCondition: not(stringEquals($Region, aws-global)) = true
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: not(isSet($Endpoint)) = true
endpoints evaluateCondition: not(stringEquals($Region, aws-global)) = true
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, false) = true
endpoints Resolving endpoint from template: {
  "url": "https://{Bucket}.s3.{Region}.{partitionResult#dnsSuffix}",
  "properties": {
    "authSchemes": [
      {
        "disableDoubleEncoding": true,
        "name": "sigv4",
        "signingName": "s3",
        "signingRegion": "{Region}"
      }
    ]
  },
  "headers": {}
}
endpoints Resolved endpoint: {
  "headers": {},
  "properties": {
    "authSchemes": [
      {
        "disableDoubleEncoding": true,
        "name": "sigv4",
        "signingName": "s3",
        "signingRegion": "eu-west-1"
      }
    ]
  },
  "url": "https://BUCKET_NAME.s3.eu-west-1.amazonaws.com/"
}
@aws-sdk/credential-provider-node defaultProvider::fromEnv
@aws-sdk/credential-provider-env fromEnv
@aws-sdk/credential-provider-node defaultProvider::fromSSO
@aws-sdk/credential-provider-node defaultProvider::fromIni
@aws-sdk/credential-provider-ini fromIni
@aws-sdk/credential-provider-ini resolveStaticCredentials
{
  clientName: 'S3Client',
  commandName: 'GetObjectCommand',
  input: { Bucket: BUCKET_NAME', Key: 'FILE_KEY' },
  error: Error: Client network socket disconnected before secure TLS connection was established
      at connResetException (node:internal/errors:721:14)
      at TLSSocket.onConnectEnd (node:_tls_wrap:1712:19)
      at TLSSocket.emit (node:events:526:35)
      at endReadableNT (node:internal/streams/readable:1408:12)
      at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
    code: 'ECONNRESET',
    path: null,
    host: 'BUCKET_NAME.s3.eu-west-1.amazonaws.com',
    port: 443,
    localAddress: undefined,
    name: 'TimeoutError',
    '$metadata': { attempts: 3, totalRetryDelay: 272 }
  },
  metadata: { attempts: 3, totalRetryDelay: 272 }
}

Expected Behavior

Retrieving the object.

It used to work some time ago, I have a clone of the same lambda of a few months back and it works.

Possible Solution

No response

Additional Information/Context

This is the current bucket policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        }
    ]
}
ennioVisco commented 3 months ago

It turns out the issue was caused by an env variable: AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1' that resulted in wrong re-use of the TLS connection.

We adopted that env variable a long time ago, probably by reading this: https://cloudash.dev/blog/http-keep-alive-lambda

github-actions[bot] commented 2 months ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.