[S3][Upload] : Wrong Location in response for file size greater than 5MB

Arun-KumarH commented 6 months ago

Describe the bug

The location returned from s3 client file upload to cloudserver is correct if the file size is less than 5MB of form <server-name>/<bucketName>/<keyName> (In client config s3ForcePathStyle is set to true). But if the file size is greater than 5MB, then the returned location is of the form <bucketName>.<server-name>/<keyName>.

This seems to happen as s3 client defaults to partSize of 5MB, but the URL / Location returned should be consistent with files greater than 5MB.

Expected Behavior

Upload result { ETag: '"a3008a5bd4094f4849e533f9bb2c5c89"', Location: 'http://127.0.0.1:8000/tenant/5MB.pdf', key: '5MB.pdf', Key: '5MB.pdf', Bucket: 'tenant' }

Current Behavior

Upload result { Location: 'http://tenant.127.0.0.1/5MB.pdf', Bucket: 'tenant', Key: '5MB.pdf', ETag: '"2ec6f604cd08eb2e091b58f666ff0a9f-2"' }

Reproduction Steps

const aws = require('aws-sdk');
const fs = require('node:fs');

const s3Client = new aws.S3({
  accessKeyId: "xxxxx",
  secretAccessKey: "xxxxx",
  endpoint: "http://127.0.0.1:8000",
  s3ForcePathStyle: true
});
async function uploadFile(bucket, key) {
  const readableStream = await fs.createReadStream('./5MB.pdf');
  const result = await new Promise((resolve, reject) => {
    s3Client.upload({
      Key: key,
      Bucket: bucket,
      Body: readableStream
    }, (err, data) => {
      if (err) {
        console.log('Error while storing object',
          {
            Key: key, Bucket: bucket, error: err, errStack: err.stack
          });
        resolve(err);
      } else {
        console.log('Object uploaded successfully');
        resolve(data);
      }
    });
  });
  console.log('Upload result', result);
}

uploadFile('tenant', '5MB.pdf').then((err, data) => {
  if (err) {
    console.log('Error', err);
  }
  console.log('Data', data);
});

Possible Solution

No response

Additional Information/Context

No response

SDK version used

aws-sdk version: 2.1621.0

Environment details (OS name and version, etc.)

NAME="openSUSE Tumbleweed" # VERSION="20240126"

aBurmeseDev commented 6 months ago

Hi @Arun-KumarH - thanks for reaching out.

I'm unable to reproduce the behavior reported so far but still looking into it. Is it consistent or intermittent? Does it only occur when endpoint is given or same for without endpoint? Any additional info would help the investigation.

Arun-KumarH commented 6 months ago

Hi @aBurmeseDev,

The issue seems to come from default part size of s3 which is 5MB. https://github.com/aws/aws-sdk-js/blob/master/lib/s3/managed_upload.js#L56

If I increase the partSize for managed_upload options as in code below to 10MB,

const result = await new Promise((resolve, reject) => {
    s3Client.upload({
      Key: key,
      Bucket: bucket,
      Body: readableStream
    }, { partSize: 10 * 1024 * 1024 }, (err, data) => {

then I get correct Location: 'http://127.0.0.1:8000/tenant/5MB.pdf', response for 5MB file.

But if I upload file bigger than 10MB, ex: 15MB with above partSize 10MB option, again we have same issue Location: 'http://tenant.127.0.0.1/15MB.pdf' where the URL is of form protocol://<bucketName>.<hostName>/<keyName> so basically when we have the file which has partSize > 1, we have this issue and it is consistent.

It occurs when the endpoint is given, if I do not provide endpoint, then we get invalid access key.

aBurmeseDev commented 2 months ago

Hi @Arun-KumarH - apologies for the delay response.

In multipart upload mode, the S3 client generates a different URL format for the Location property in the response. This format is <bucketName>.<endpointUrl>/<keyName> which is consistent with the behavior you're observing for files larger than 5 MB.

The reason for this difference is that the S3 client follows the AWS S3 API specification which defines different URL formats for single-part and multipart uploads. For single-part uploads (files smaller than the partSize), the Location URL follows the format you expect: <endpointUrl>/<bucketName>/<keyName>. However, for multipart uploads, the Location URL format changes to <bucketName>.<endpointUrl>/<keyName>.

While this behavior may seem inconsistent, it is intentional and follows the AWS S3 API specification. SDK team made this design decision to align with the S3 API and maintain compatibility with the AWS S3 service.

If you require a consistent URL format for both single-part and multipart uploads, you could consider implementing a custom function that constructs the desired URL format based on the upload type.

async function uploadFile(bucket, key) {
  const readableStream = await fs.createReadStream('./5MB.pdf');
  const result = await new Promise((resolve, reject) => {
    s3Client.upload({
      Key: key,
      Bucket: bucket,
      Body: readableStream
    }, (err, data) => {
      if (err) {
        console.log('Error while storing object',
          {
            Key: key, Bucket: bucket, error: err, errStack: err.stack
          });
        resolve(err);
      } else {
        console.log('Object uploaded successfully');
        const location = `http://127.0.0.1:8000/${bucket}/${key}`;
        data.Location = location;
        resolve(data);
      }
    });
  });
  console.log('Upload result', result);
}

Hope it helps! Best, John

aws / aws-sdk-js