search

girder / girder_web_components

Reusable Javascript and VueJS components for interacting with a Girder server.
https://gwc.girder.org
Apache License 2.0
16 stars 9 forks source link

[BUG] [Girder-Upload] Upload doesn't tolerate very large numbers of files. #273

Closed subdavis closed 3 years ago

subdavis commented 4 years ago

I tried to upload 2K at once. Upload threw a resource exceeded exception. I think we need to limit the number of outstanding POST requests.

Even tallying the files to provide pre-upload summary takes too long.

zachmullen commented 4 years ago

Upload threw a resource exceeded exception

Is this a server side or client side exception? Could you provide the error message?

I intend to use a pool-based approach moving forward, so this wouldn't be an issue anymore.

manthey commented 4 years ago

I don't know if this is related, but in another project I tried to fetch a huge number of small files at once and I got a similar error. Chrome fails in odd ways (using both HTTP 1.1 and HTTP 2.0), and not always at the same point in the process. Bluebird has a nice method to limit concurrent requests, and I ended using that with a concurrency limit that was ~24. In HTTP 1.1 most browsers will limit concurrency to 6, but there is overhead to start and stop. I ended up with the best throughput by using roughly between 2 and 5 times that number.

subdavis commented 4 years ago

Is this a server side or client side exception? Could you provide the error message?

Client error. I believe too many outstanding requests are allocated at once. This isn't about concurrent connections to girder, but rather whatever memory Chrome allocates when an XMLHTTPRequest is constructed (probably)

Screenshot from 2020-09-01 14-52-45

I intend to use a pool-based approach moving forward, so this wouldn't be an issue anymore.

Yeah, we should do something similar here.

subdavis commented 3 years ago

This was resolved in #283 and #284