Cannot upload 8000 files simultaneously

[frankrousseau]

A user complained that he wasn't able to upload his picture folder of 8000 files (his whole Cozy crashed).

We should (at least):

• Prevent the whole Cozy to crash in that kind of situation
• Prevent the file app to crash
• Add a message telling that the app cannot handle so many files

[clochix]

Do we know why the server crashed ? Is it related to the number of files, or to the total size ?

[frankrousseau]

Sorry, I don't know. Is that something you can reproduce?

[clochix]

So, I created 10.000 files of only 1o and tried to upload them. Firefox crashed after ± 1:30, after eating all my memory. There seems to be some memory leak on frontend (to be confirmed, maybe is it in the browser, not in our code).

After 1 h 30, only 1300 files where uploaded (I tried on my local VM, so this is not a network issue). Not that much.

Looking at CouchDB logs, I see 26 GET request (in around 2s) for every uploaded file !

So I think there's probably some place for improvements ;-)

(FTR: to create the 10.000 files, I used this command : for i in {1..10000}; do dd if=/dev/urandom bs=1 count=1 of=file$i; done) )

[clochix]

BTW, congrats @jsilvestre for the work you did on displaying folder with lot of files : it works perfectly \o/

[jsilvestre]

I can confirm that uploading large collection of elements is not efficient, on our side. There are constraints, like detecting if there are files to overwrite, but here are some inputs:

• we should check that when elements are added to the collection, it doesn't trigger a sort for each element (I had the case working on rendering big lists). It's easy to see with dev tools.
• we create one model in the collection for each element. That's a big local cache, there are probably way to improve things here, but it's not trivial with the current UX (elements being uploaded or waiting for being uploaded are shown in the list)

[clochix]

There are probably also some optimization to perform on server side. I can confirm that when I upload a single file, I see 27 requests for this file in the logs of CouchDB (event if the indexer is stopped).

[aenario]

Some optimizations are in the DS :

• files create File#66
  • DS POST file#66
  • event file.create
  • DS GET file#66
  • Files GET file#66
• files index File#66
  • DS GET File#66
• files add binary to File#66
  • DS GET File#66
  • DS PUT File#66
  • Event file.update
  • DS GET File#66(x2 - https://github.com/cozy/cozy-data-system/blob/eca4847ba3cb94030405e5120e5db822de41ab46/server/lib/feed.coffee#L131)
  • Files GET File#66 (realtime)
• files add checksum to File#66
  • DS GET File#66
  • DS PUT File#66
  • Event file.update
  • DS GET File#66(x2)
  • Files GET File#66 (realtime)
• files get file#66 (to get binary info) https://github.com/cozy/cozy-files/blob/6c11cd541a78c1e50dd8724fa369895a7386afc7/server/controllers/files.coffee#L265

Not sure where the 17 others come from :-/