search

transloadit / uppy

The next open source file uploader for web browsers :dog:
https://uppy.io
MIT License
28.98k stars 2k forks source link

Question: Is there a way to allow Uppy to send many files in only one request? #3632

Closed francoGiulianini closed 2 years ago

francoGiulianini commented 2 years ago

Hi! I'm currently working on a project with React on the front and NestJS on the backend. We're trying to solve a performance improvement issue related to uploading a massive amount of pictures to S3 (Thousands of photos).

The thing is that we need to do some previous processing before uploading them: We have to create a watermark version of the photos, a thumbnail version, and upload both of them plus the original one to S3, after saving some of the metadata into our database, and there's also some facial recognition processing involved. Due to the nature of the project, it is important to do all these things upon uploading the picture, so trying to implement a post-upload processing module would be our last choice.

I checked @uppy/aws-s3, but I understand that in this case this is not useful, since it goes straight to S3 without interacting with a backend endpoint.

So, I was wondering if Uppy is able to hit an endpoint sending a list of n photos at once, instead of hitting the same endpoint n times with one photo at the time. Doing that, we would be allowed to handle the list as a whole in the backend, improving queries and upload performance. Any help or tip would be really appreciated. Thanks in advance!

Murderlon commented 2 years ago

Hi, uploading strategies carefully determine how to send files and you can't control it to the extend of configuring how many files to send in one request. That would be error-prone and it depends on the file sizes as well.

For you use case, I recommend the following things:

  1. Do your pre-processing with addPreProcessor in custom plugins. You can take a look at @uppy/compressor as an example.
  2. Use @uppy/tus with either a hosted (via Transloadit) or self-hosted Tus server. This has some benefits:
    1. Reliable, resumable uploads
    2. Exponential backoff for large amount of files. If you are letting users upload thousands of files it might overload your server (or as protection against DoS). @uppy/tus has exponential backoff built-in. Meaning if your server (or proxy) returns HTTP 429 because it's being overloaded, @uppy/tus will find the ideal sweet spot to keep uploading without overloading
    3. Tus servers, such as tusd, also have an S3 store so you can pass the files to S3 without writing that code yourself.

However, @uppy/aws-s3 (or aws-s3-multipart) might also be enough for you without hosting an extra server.

arturi commented 2 years ago

If you want simple uploads, you could use @uppy/xhr-upload with bundle: true, it will send all files in a single multipart request.