search

scireum / s3ninja

S3 ninja emulates the Amazon S3 API for developement and test purposes
MIT License
493 stars 91 forks source link

Hangs on parsing the Body #228

Closed nkalinov closed 1 year ago

nkalinov commented 1 year ago

As soon as I try to parse the Body stream using various methods the process hangs. My sample code:

const s3Client = new S3Client({ endpoint: 'http://localhost:9000' });

const { Body } = await s3Client.send(
  new GetObjectCommand({
    Key: '...',
    Bucket: '...',
  })
);

const bodyAsString = await Body.transformToString();
console.log('bodyAsString', bodyAsString);

It doesn't throw nor it gets to console.log, it just hangs. I also can't see any activity in the Access Logs. Given the above I am not even sure if it hits the container. I have mapped ui-shell.localhost to 127.0.0.1 in /etc/host

To help debugging I enabled logger on the S3 client and here's the output after the above GetObjectCommand:

endpoints Initial EndpointParams: {
  "Bucket": "ui-shell",
  "ForcePathStyle": false,
  "UseArnRegion": false,
  "DisableMultiRegionAccessPoints": false,
  "Accelerate": false,
  "UseGlobalEndpoint": false,
  "UseFIPS": false,
  "Endpoint": "http://localhost:9000/",
  "Region": "us-east-1",
  "UseDualStack": false
}
endpoints evaluateCondition: isSet($Region) = true
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: substring($Bucket, 49, 50, true) = null
endpoints evaluateCondition: isSet($Bucket) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: not(isSet(parseURL($Endpoint))) = false
endpoints evaluateCondition: isSet($ForcePathStyle) = true
endpoints evaluateCondition: booleanEquals($ForcePathStyle, true) = false
endpoints evaluateCondition: aws.isVirtualHostableS3Bucket($Bucket, false) = true
endpoints evaluateCondition: aws.partition($Region) = {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "US East (N. Virginia)"
}
endpoints assign: partitionResult := {
  "dnsSuffix": "amazonaws.com",
  "dualStackDnsSuffix": "api.aws",
  "name": "aws",
  "supportsDualStack": true,
  "supportsFIPS": true,
  "description": "US East (N. Virginia)"
}
endpoints evaluateCondition: isValidHostLabel($Region, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: booleanEquals($Accelerate, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals(getAttr($url, isIp), true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals(getAttr($url, isIp), false) = true
endpoints evaluateCondition: stringEquals($Region, aws-global) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals(getAttr($url, isIp), true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals(getAttr($url, isIp), false) = true
endpoints evaluateCondition: stringEquals($Region, aws-global) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals(getAttr($url, isIp), true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals(getAttr($url, isIp), false) = true
endpoints evaluateCondition: not(stringEquals($Region, aws-global)) = true
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals(getAttr($url, isIp), true) = false
endpoints evaluateCondition: booleanEquals($UseDualStack, false) = true
endpoints evaluateCondition: booleanEquals($UseFIPS, false) = true
endpoints evaluateCondition: booleanEquals($Accelerate, false) = true
endpoints evaluateCondition: isSet($Endpoint) = true
endpoints evaluateCondition: parseURL($Endpoint) = {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints assign: url := {
  "scheme": "http",
  "authority": "localhost:9000",
  "path": "/",
  "normalizedPath": "/",
  "isIp": false
}
endpoints evaluateCondition: booleanEquals(getAttr($url, isIp), false) = true
endpoints evaluateCondition: not(stringEquals($Region, aws-global)) = true
endpoints evaluateCondition: booleanEquals($UseGlobalEndpoint, false) = true
endpoints Resolving endpoint from template: {
  "url": "{url#scheme}://{Bucket}.{url#authority}{url#path}",
  "properties": {
    "authSchemes": [
      {
        "name": "sigv4",
        "signingRegion": "{Region}",
        "signingName": "s3",
        "disableDoubleEncoding": true
      }
    ]
  },
  "headers": {}
}
endpoints Resolved endpoint: {
  "headers": {},
  "properties": {
    "authSchemes": [
      {
        "name": "sigv4",
        "signingRegion": "us-east-1",
        "signingName": "s3",
        "disableDoubleEncoding": true
      }
    ]
  },
  "url": "http://ui-shell.localhost:9000/"
}
{
  clientName: 'S3Client',
  commandName: 'GetObjectCommand',
  input: { Key: 'manifest.json', Bucket: 'ui-shell' },
  output: {
    Metadata: {},
    Body: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 1,
      _maxListeners: undefined,
      socket: [Socket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: false,
      rawHeaders: [Array],
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 200,
      statusMessage: 'Connection Established',
      client: [Socket],
      _consuming: false,
      _dumped: false,
      req: [ClientRequest],
      transformToByteArray: [AsyncFunction: transformToByteArray],
      transformToString: [AsyncFunction: transformToString],
      transformToWebStream: [Function: transformToWebStream],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 2,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0,
      [Symbol(RequestTimeout)]: undefined
    }
  },
  metadata: {
    httpStatusCode: 200,
    requestId: undefined,
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}

Body is IncomingMessage - I guess that's why Body.transformToString() doesn't work? Any ideas what could be wrong?

jakobvogel commented 1 year ago

Hello @nkalinov 👋 Thanks for getting in touch. Unfortunately, I am not an expert in the library you are using. Can you please provide a repository with a minimal example? In this case, I could have a look myself.

nkalinov commented 1 year ago

I'm using the latest official client - "@aws-sdk/client-s3": "^3.301.0" Here's a simple repo with the script that I'm trying, hope that's useful - https://github.com/nkalinov/issue-repro-s3ninja It prints "Body has transformToString method: true" and then hangs.

jakobvogel commented 1 year ago

Hello @nkalinov! I just saw by coincidence that you have updated your comment with a link to an example – many thanks! And sorry for my late reaction, I do not get notifications for edits. In any case, I will have a look and get back to you asap.

jakobvogel commented 1 year ago

Hi @nkalinov! I just happened to have time to check the reported issue. Your example did not work out of the box for me. I fixed it by adding a region and credentials to the configuration struct:

import {GetObjectCommand, S3Client} from '@aws-sdk/client-s3';
import {fromIni} from '@aws-sdk/credential-providers';

const s3Client = new S3Client({
    endpoint: 'http://localhost:9444',
    region: 'us-east-1',
    credentials: fromIni({profile: 's3ninja'}),
    forcePathStyle: true,
    ...

Using the latest versions of the SDK, ^3.353.0, all looks good here. I run S3 Ninja directly, outside of Docker, and I made sure that the object I am querying actually exists.

So, unfortunately, I do not see a way a change in this project could help you resolve the issue. I am sorry. 😔

Please reopen the issue if you have further questions, remarks, etc.