search

datopian / r2-bucket-uploader

Cloudflare R2 bucket File Uploader with multipart upload enabled. Tested with files up to 10 GB size.
MIT License
85 stars 11 forks source link

Multipart upload fails for large files #6

Open sorin89 opened 7 months ago

sorin89 commented 7 months ago

Tested with 5GB and 8GB files. Works fine for a 2GB file.

sorin89 commented 7 months ago

I realized what the problem is. When the file is too large, it gets stuck on creating the arrayBuffer and fileHash.

anuveyatsu commented 6 months ago

Interesting @sorin89 - computation of file hash would happen locally so I wouldn't expect that being a problem but good to know. I believe we should make it optional.

eltonvilanculo commented 6 months ago

I tried using the solution in Nextjs 14, however I'm hvaing 405 error.

eltonvilanculo commented 6 months ago

I tried using the solution in Nextjs 14, however I'm hvaing 405 error.

Solved, it was related to new update of Nextjs issue

anuveyatsu commented 1 month ago

hi @sorin89 have you tried with removing the hashing function? Please let us know if it helped.

also @steveoni would you mind commenting with your latest tests - I know that you've been playing with the multi-part upload recently 😄. We'd be interested in what's the largest size you tried to upload, what is the issues (if any) and did you remove the hashing function?

steveoni commented 1 month ago

hi @sorin89 have you tried with removing the hashing function? Please let us know if it helped.

also @steveoni would you mind commenting with your latest tests - I know that you've been playing with the multi-part upload recently 😄. We'd be interested in what's the largest size you tried to upload, what is the issues (if any) and did you remove the hashing function?

It’s true that for large file , generating the hash string breaks the app. So I tried with a fixed hash string and was able to upload a 10gb file.

So the solution will be to find other alternatives to generate hash string

anuveyatsu commented 1 month ago

@steveoni I would say we don't need to generate file hash at all by default. Hashing is useful if you would like to have:

  1. Versioning so that md5 hash can be used as a signature of the file content.
  2. Save storage, i.e., the same file content wouldn't be stored multiple times.

I think the main purpose of this library is to provide a basic app to be able to upload files from local disk to a bucket.

Could you please open a pull request removing hashing function? We can use file name as a prefix in the bucket.