error: runtime: out of memory

[GoogleCodeExporter]

I issued this command
 ~/backup/camlistore/bin/camput file --permanode --title="Pictures" Pictures
on a directory of jpg photos about 7.5GB

about 3GB into the creation of the blobs I got this error

.......
2014/03/30 19:41:03 Received blob 
[sha1-db75b1b14526eeec6973ef71fd0cf4ee808780cb; 70673 bytes]
fatal error: runtime: out of memory

goroutine 137828 [running]:
runtime.throw(0x1689697)
        /usr/lib/go/src/pkg/runtime/panic.c:464 +0x69 fp=0x7f964c0c4f08
runtime.SysMap(0xc317730000, 0x100000000, 0x16a06b8)
        /usr/lib/go/src/pkg/runtime/mem_linux.c:131 +0xfe fp=0x7f964c0c4f38
....

The rest of the output is in the attached file.
I am running go version go1.2.1 linux/amd64

When I try to restart camlistored I get

barryb@edge:~/backup/camlistore/bin$ ./camlistored 
2014/03/30 21:20:29 Starting camlistored version 2014-03-27-768cb6e; Go go1.2.1 
(linux/amd64)
2014/03/30 21:20:29 Using config file 
/home/barryb/.config/camlistore/server-config.json
2014/03/30 21:20:29 Starting to listen on http://localhost:3179
2014/03/30 21:20:30 Caught panic installer handlers: error instantiating 
storage for prefix "/index-kv/", type "kvfileindexer": error opening 
/home/barryb/var/camlistore/camli-index.kvdb: unexpected EOF
Error parsing config: Caught panic: error instantiating storage for prefix 
"/index-kv/", type "kvfileindexer": error opening 
/home/barryb/var/camlistore/camli-index.kvdb: unexpected EOF

Bill 

Original issue reported on code.google.com by waba...@gmail.com on 31 Mar 2014 at 4:25

Attachments:

• camlistorecrash

[GoogleCodeExporter]

Original comment by bradfitz on 21 Oct 2014 at 2:48

• Changed state: HelpWanted
• Added labels: Performance

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I ran across something similar a while ago, possibly the same problem.

I guess that this information is better than none, even if may not be entirely 
accurate.

After some testing I found that the import mostly went fine, but the memory 
consumption increased when certain files were imported. If i remember 
correctly, i got the same surge in memory usage even if i just imported one of 
the problematic files alone, so the problem didn't seem to be related to 
importing large amounts of files at once.

I could easily import many files, as long as none of the problematic ones were 
included.

My conclusion (by comparing good and bad files) was that the problem may have 
been directly related to malformed EXIF-data. I think at least some of the 
files were produced by the built in camera app on an HTC Desire S.

Again, this is just from memory. I don't have access to the files right now, 
but I'll try to re-create the problem, and update the issue when I have more 
information.

Original comment by ola.siks...@gmail.com on 29 Nov 2014 at 6:36

[GoogleCodeExporter]

This issue has moved to https://camlistore.org/issue/414

Original comment by bradfitz on 14 Dec 2014 at 11:37

• Changed state: Moved
• Added labels: IssueMoved, Restrict-AddIssueComment-Commit