web(?): Dropping a lot of files breaks stuff (too many open files)

[GoogleCodeExporter]

Sorry for the long description.

As a test I set up a fresh Camlistore instance and dragged a folder filled with 
~2000 64x64 png icons to the WebUI.

The webui froze and stayed frozen for about 2 minutes, after which I saw the 
blobs come in in camlistore console output. 

After some time messages akin to:
2014/02/13 15:41:35 filename "star.png" exif = 0001-01-01 00:00:00 +0000 UTC, 
All methods failed to find a creation time or modtime.
began to appear, which isn't that strange either considering the files don't 
contain exif data as far as I'm aware.

A few minutes later however, messages like these appeared a few thousand times:
2014/02/13 15:32:14 http: Accept error: accept tcp [::]:80: too many open 
files; retrying in 20ms

At the same time the WebUI gave an alert regarding an unexpected result. 
Stupidly I haven't noted it. It mentioned a blobRef and a Permanode, one of 
which was missing.

Refreshing the page shows me that exactly 50 files have made it in and 12 are 
missing their thumbnail and thus show up as "Unknown Title".
In the console I see quite a few messages like:
2014/02/13 15:35:22 index.GetFileInfo(file 
sha1-4c9a48fbdaa2bf26590ed5ce40fd2408dbf77f83) failed; index stale?

Dropping the same files onto the webui again freezes the ui again but (after 
about 15.000 logged requests before Chrome inspector craps out) does complete 
the upload of all files.

I'm using EC2, a micro instance in this case.

Original issue reported on code.google.com by gov...@ver.slu.is on 13 Feb 2014 at 3:58

[GoogleCodeExporter]

Is this on tip or the 0.6 release? The hung ui should not occur on tip.

Original comment by zbo...@gmail.com on 13 Feb 2014 at 4:06

[GoogleCodeExporter]

This was on tip (I assume, master on camlistore.googlesource.com/camlistore)

(mobile)
Op 13 feb. 2014 17:06 schreef <camlistore@googlecode.com>:

Original comment by gov...@ver.slu.is on 13 Feb 2014 at 4:09

[GoogleCodeExporter]

Well I see at least one thing that might be causing the blocked UI.

Unclear if that's the only bug you are seeing. Can you try uploading a smaller 
number of files and report back whether that works as expected?

Original comment by zbo...@gmail.com on 14 Feb 2014 at 3:28

• Changed state: Started

[GoogleCodeExporter]

Can you try again after 17e3799cdaf08fc00236ba6d4e29f0db9d8d9430 and let me 
know if it's any better?

Original comment by zbo...@gmail.com on 15 Feb 2014 at 3:21

[GoogleCodeExporter]

After 17e3799cdaf08fc00236ba6d4e29f0db9d8d9430 it is much better and I do not 
receive any of the errors mentioned above.

The browser still freezes initially when the 2600 files are dropped, before 
Camlistore receives anything. This lasts exactly a minute and a half.
Once the "Received blob" messages start scrolling in Camlistore the browser 
unfreezes (unlike before) and the UI is usable and responsive while the files 
are uploading in the background. The new files do not appear in the WebUI until 
I refresh the page though, but that should probably be a separate issue?

Original comment by gov...@ver.slu.is on 17 Feb 2014 at 9:04

[GoogleCodeExporter]

I don't understand the commit message for 
17e3799cdaf08fc00236ba6d4e29f0db9d8d9430.  It says "Turn Closure's DEBUG flag 
off." but the commit sets it to true.

Original comment by bradfitz on 17 Feb 2014 at 6:49

[GoogleCodeExporter]

In that case I don't understand why my experience was better after that commit.
I'll try to reproduce my issues with the commit before that and the new one 
which actually turns debugging off.

Original comment by gov...@ver.slu.is on 18 Feb 2014 at 8:03

[GoogleCodeExporter]

No recent info. Closing for now. Please reopen if relevant.

Original comment by bradfitz on 21 Oct 2014 at 2:33

• Changed state: TimedOut

[GoogleCodeExporter]

This issue has moved to https://camlistore.org/issue/381

Original comment by bradfitz on 14 Dec 2014 at 11:37

• Added labels: IssueMoved, Restrict-AddIssueComment-Commit