error: runtime: out of memory

[camlistorebot]

by wabarry:

I issued this command
 ~/backup/camlistore/bin/camput file --permanode --title="Pictures" Pictures
on a directory of jpg photos about 7.5GB

about 3GB into the creation of the blobs I got this error

.......
2014/03/30 19:41:03 Received blob [sha1-db75b1b14526eeec6973ef71fd0cf4ee808780cb; 70673
bytes]
fatal error: runtime: out of memory

goroutine 137828 [running]:
runtime.throw(0x1689697)
        /usr/lib/go/src/pkg/runtime/panic.c:464 +0x69 fp=0x7f964c0c4f08
runtime.SysMap(0xc317730000, 0x100000000, 0x16a06b8)
        /usr/lib/go/src/pkg/runtime/mem_linux.c:131 +0xfe fp=0x7f964c0c4f38
....

The rest of the output is in the attached file.
I am running go version go1.2.1 linux/amd64

When I try to restart camlistored I get

barryb@edge:~/backup/camlistore/bin$ ./camlistored 
2014/03/30 21:20:29 Starting camlistored version 2014-03-27-768cb6e; Go go1.2.1
(linux/amd64)
2014/03/30 21:20:29 Using config file /home/barryb/.config/camlistore/server-config.json
2014/03/30 21:20:29 Starting to listen on http://localhost:3179
2014/03/30 21:20:30 Caught panic installer handlers: error instantiating storage for
prefix "/index-kv/", type "kvfileindexer": error opening
/home/barryb/var/camlistore/camli-index.kvdb: unexpected EOF
Error parsing config: Caught panic: error instantiating storage for prefix
"/index-kv/", type "kvfileindexer": error opening
/home/barryb/var/camlistore/camli-index.kvdb: unexpected EOF

Bill

Attachments:

1. camlistorecrash (55376 bytes)

[bradfitz]

Comment 1:

Labels changed: added performance.

Status changed to HelpWanted.

[camlistorebot]

Comment 2 by ola.sikstrom:

I ran across something similar a while ago, possibly the same problem.
I guess that this information is better than none, even if may not be entirely accurate.
After some testing I found that the import mostly went fine, but the memory consumption
increased when certain files were imported. If i remember correctly, i got the same
surge in memory usage even if i just imported one of the problematic files alone, so the
problem didn't seem to be related to importing large amounts of files at once.
I could easily import many files, as long as none of the problematic ones were included.
My conclusion (by comparing good and bad files) was that the problem may have been
directly related to malformed EXIF-data. I think those were produced by a 
Again, this is just from memory. I don't have access to the files right now, but I'll
try to re-create the problem, and update the issue when I have more information.

[camlistorebot]

Comment 3 by ola.sikstrom:

I ran across something similar a while ago, possibly the same problem.
I guess that this information is better than none, even if may not be entirely accurate.
After some testing I found that the import mostly went fine, but the memory consumption
increased when certain files were imported. If i remember correctly, i got the same
surge in memory usage even if i just imported one of the problematic files alone, so the
problem didn't seem to be related to importing large amounts of files at once.
I could easily import many files, as long as none of the problematic ones were included.
My conclusion (by comparing good and bad files) was that the problem may have been
directly related to malformed EXIF-data. I think at least some of the files were
produced by the built in camera app on an HTC Desire S.
Again, this is just from memory. I don't have access to the files right now, but I'll
try to re-create the problem, and update the issue when I have more information.

[hullerob]

I also encountered this issue. Looks like rwcarlsen/goexif/issues/20 and rwcarlsen/goexif/issues/30

This is fixed by https://github.com/rwcarlsen/goexif/commit/41dad3aa083329f3f672b7095a9ea8a0c384bbe8

Applying that to camlistore.org/third_party/github.com/rwcarlsen/goexif fixes issue for me.

[mpl]

@hullerob thanks for the tip!

https://camlistore-review.googlesource.com/4826

[mpl]

fixed by 6c2c5811794237d5be32236c5c8ff9c652a39c8d