Repeated bucket name in req.file.location - Seems an AWS issue

[chuckntaylor]

Just a heads up, I was writing out some unit tests today and came across some odd behavior where the bucket name would be repeated in the req.file.location.

For example:

{
  "bucket": "my-bucket",
  "key": "image-folder/some-file-name.png",
  "location": "https://my-bucket.my-bucket.s3.us-east-1.amazonaws.com/image-folder/some-file-name.png",
  ...
}

This appears to be a problem with @aws-sdk/client-s3 and @aws-sdk/util-endpoints

Last week, I had an issue with @aws-sdk/client-s3 https://github.com/aws/aws-sdk-js-v3/issues/4060 where it started saying that it cannot find @aws-sdk/util-endpoints. They have since said the issue is resolved. However, when I use the latest @aws-sdk/client-s3: ^3.196.0 I have the problem mentioned above.

When I roll back to @aws-sdk/client-s3: 3.180.0 all is good.

I just want to bring this to your attention. I was unsure how best to report this to AWS as I wasn't sure where the result callback information for location is generated.

If you do end up filing an issue with aws, I would love to follow it.

[AnnSamsonenko]

I'm facing the same problem and didn't find the way how to fix it but only to cut a duplicate of bucket name in the file.location by myself. But it seems to happen not with every file I upload, some of them duplicates, and some do not (I can't figure out this strange pattern😅) so I really don't know what to do with this

[chuckntaylor]

To be honest, I have kept @aws-sdk/client-s3 pinned to 3.180.0 in my package.json for the time being, and haven't experimented since my earlier posting. However, I think your thought is a good one for now — search the resulting string and remove a duplicate bucket name if it exists. Maybe something like:

const location = req.file.location.replace(`${bucketName}.${bucketName}.`, `${bucketName}.`)

[tusbar]

The issue is related to a mismatch with shared dependencies between @aws-sdk/client-s3 (or probably some other aws-sdk clients) and @aws-sdk/lib-storage. I used to get this error as well, I am now getting

TypeError: this.client.config.endpoint is not a function
    at Upload.__uploadUsingPut (/path/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:48:32)
    at Upload.__doConcurrentUpload (/path/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:91:39)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Promise.all (index 0)
    at async Upload.__doMultipartUpload (/path/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:141:9)
    at async Upload.done (/path/node_modules/@aws-sdk/lib-storage/dist-cjs/Upload.js:37:16)

I think the correct way would be to change @aws-sdk/lib-storage to be a peer-dependency as well. In the meantime, you can easily fix the issue by bumping the version of @aws-sdk/lib-storage (either remove/add multer-s3 as the range is very permissive, or use something like yarn resolutions).

[AnnSamsonenko]

sincere thanks to you guys!