[S3][Upload] : Wrong Location in response for file size greater than 5MB

[Arun-KumarH]

Describe the bug

The location returned from s3 client file upload to cloudserver is correct if the file size is less than 5MB of form <server-name>/<bucketName>/<keyName> (In client config s3ForcePathStyle is set to true). But if the file size is greater than 5MB, then the returned location is of the form <bucketName>.<server-name>/<keyName>.

This seems to happen as s3 client defaults to partSize of 5MB, but the URL / Location returned should be consistent with files greater than 5MB.

Expected Behavior

Upload result { ETag: '"a3008a5bd4094f4849e533f9bb2c5c89"', Location: 'http://127.0.0.1:8000/tenant/5MB.pdf', key: '5MB.pdf', Key: '5MB.pdf', Bucket: 'tenant' }

Current Behavior

Upload result { Location: 'http://tenant.127.0.0.1/5MB.pdf', Bucket: 'tenant', Key: '5MB.pdf', ETag: '"2ec6f604cd08eb2e091b58f666ff0a9f-2"' }

Reproduction Steps

const aws = require('aws-sdk');
const fs = require('node:fs');

const s3Client = new aws.S3({
  accessKeyId: "xxxxx",
  secretAccessKey: "xxxxx",
  endpoint: "http://127.0.0.1:8000",
  s3ForcePathStyle: true
});
async function uploadFile(bucket, key) {
  const readableStream = await fs.createReadStream('./5MB.pdf');
  const result = await new Promise((resolve, reject) => {
    s3Client.upload({
      Key: key,
      Bucket: bucket,
      Body: readableStream
    }, (err, data) => {
      if (err) {
        console.log('Error while storing object',
          {
            Key: key, Bucket: bucket, error: err, errStack: err.stack
          });
        resolve(err);
      } else {
        console.log('Object uploaded successfully');
        resolve(data);
      }
    });
  });
  console.log('Upload result', result);
}

uploadFile('tenant', '5MB.pdf').then((err, data) => {
  if (err) {
    console.log('Error', err);
  }
  console.log('Data', data);
});

Possible Solution

No response

Additional Information/Context

No response

SDK version used

aws-sdk version: 2.1621.0

Environment details (OS name and version, etc.)

NAME="openSUSE Tumbleweed" # VERSION="20240126"

[aBurmeseDev]

Hi @Arun-KumarH - thanks for reaching out.

I'm unable to reproduce the behavior reported so far but still looking into it. Is it consistent or intermittent? Does it only occur when endpoint is given or same for without endpoint? Any additional info would help the investigation.

[Arun-KumarH]

Hi @aBurmeseDev,

The issue seems to come from default part size of s3 which is 5MB. https://github.com/aws/aws-sdk-js/blob/master/lib/s3/managed_upload.js#L56

If I increase the partSize for managed_upload options as in code below to 10MB,

const result = await new Promise((resolve, reject) => {
    s3Client.upload({
      Key: key,
      Bucket: bucket,
      Body: readableStream
    }, { partSize: 10 * 1024 * 1024 }, (err, data) => {

then I get correct Location: 'http://127.0.0.1:8000/tenant/5MB.pdf', response for 5MB file.

But if I upload file bigger than 10MB, ex: 15MB with above partSize 10MB option, again we have same issue Location: 'http://tenant.127.0.0.1/15MB.pdf' where the URL is of form protocol://<bucketName>.<hostName>/<keyName> so basically when we have the file which has partSize > 1, we have this issue and it is consistent.

It occurs when the endpoint is given, if I do not provide endpoint, then we get invalid access key.