search

anacronw / multer-s3

multer storage engine for amazon s3
MIT License
660 stars 190 forks source link

Large files getting relative url #93

Closed sarahbethfederman closed 3 years ago

sarahbethfederman commented 6 years ago

I'm having a strange issue when dealing with large (~55MB) file uploads. In the callback after upload, I'm saving the file location to my db like so req.files.sourceFile[0].location

Usually this resolves to an absolute path, but on really large files this is resolving to a relative path that consists of <bucket-name>/<file-key>/. Is there some async issue I'm unaware of? The file itself is being uploaded and exists on s3 like it's supposed to, I'm just not getting the absolute url in the callback.

tiagomsmagalhaes commented 6 years ago

If files are short location key displays https://bucketlocation.tld/somefile.extension but as I tried to upload a 8.6 Mb video I only get bucketlocation.tldas value for location

@badunk

sarahbethfederman commented 6 years ago

Just hit this again with a file at 34.3MB so I guess it starts happening around there. Anything I can do to help this along @badunk? I think it may be a bit beyond my current skill level to solve but I can try and look into it

guliash commented 6 years ago

I think this issue is not related to multer-s3. @sarahbethfederman, @tiagomsmagalhaes are you using Amazon S3? Or maybe DigitalOcean Spaces?

sarahbethfederman commented 6 years ago

I'm using the enterprise s3 (an internal IT-managed instance with our own endpoint). I wrote a check that changes any relative URLs to a full one that's been fine for the time being (since they're still being uploaded successfully, just the URL is wrong)

tiagomsmagalhaes commented 6 years ago

@guliash I’m using digital ocean spaces.

guliash commented 6 years ago

It seems that aws-sdk manually adds scheme for a single part upload (when file is small) https://github.com/aws/aws-sdk-js/blob/master/lib/s3/managed_upload.js#L662. But for large files it does nothing https://github.com/aws/aws-sdk-js/blob/master/lib/s3/managed_upload.js#L625.

@tiagomsmagalhaes According to the docs Amazon S3 returns full location in case of multipart upload. But DO Spaces does not return full location in case of multipart upload (see here).

So I think the problem is in different responses from Amazon S3 and DO Spaces.

@sarahbethfederman Can it be that the instance's responses in case of Complete Multipart Upload operation do not contain full location?

LinusU commented 3 years ago

Seems like this should be reported upstream to the AWS sdk, if it's not already fixed...

Closing due to inactivity...