mem usage on camlistored is too damn high, with long camput. probably leak.

[GoogleCodeExporter]

Note this is on a machine with 16GB ram.

devcam server -wipe
devcam put file /some/big/dir/*

top - 18:47:40 up  5:12,  3 users,  load average: 8.70, 9.01, 7.84
Tasks: 236 total,   1 running, 235 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.5 us,  0.2 sy,  0.0 ni, 78.0 id, 21.4 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  16389532 total, 16152668 used,   236864 free,    14496 buffers
KiB Swap:        0 total,        0 used,        0 free,    71860 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 5910 mpl       20   0 15.2g  14g  488 S   1.0 90.8  10:29.40 camlistored
 5975 mpl       20   0 3709m 362m 1220 S   1.0  2.3   3:55.81 camput
 6442 mpl       20   0 20664 1692 1112 R   0.3  0.0   0:00.04 top
    1 root      20   0 27164 1548    4 S   0.0  0.0   0:01.11 init
    2 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kthreadd

The memory does not climb that high immediately. I was able to reproduce it in 
the same conditions.

Original issue reported on code.google.com by mathieu....@gmail.com on 8 Nov 2013 at 7:37

[GoogleCodeExporter]

Note, you should be able to run 'CAMLI_HTTP_PPROF=1 devcam server ...' and then 
point 'go pprof tool http://localhost:3179/debug/pprof/heap' if you want more 
insight into this.

Original comment by couchmo...@gmail.com on 8 Nov 2013 at 7:41

[GoogleCodeExporter]

Good point, thanks.
One funny thing to remember in that case is too run pprof soon enough, before 
you actually run out of mem, otherwise, blamo:

mpl@valisk:~$ go tool pprof http://localhost:3179/debug/pprof/heap
runtime: panic before malloc heap initialized
fatal error: runtime: cannot allocate heap metadata

attached is an svg made with the web command, when there was still enough mem

Original comment by mathieu....@gmail.com on 11 Nov 2013 at 4:26

Attachments:

• pprof7650.0.svg

[GoogleCodeExporter]

Can you try patching in https://camlistore.org/r/1017/ and see how much it 
helps?

Original comment by couchmo...@gmail.com on 11 Nov 2013 at 5:07

[GoogleCodeExporter]

Nice! waiting for the big camput to finish, but this looks promising indeed.

Original comment by mathieu....@gmail.com on 11 Nov 2013 at 6:23

[GoogleCodeExporter]

with that patch:

top - 19:29:50 up  5:47,  5 users,  load average: 5.47, 5.24, 4.95
Tasks: 242 total,   2 running, 240 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.1 us,  1.5 sy,  0.0 ni, 74.8 id, 19.5 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem:  16389532 total, 16168692 used,   220840 free,   415092 buffers
KiB Swap:        0 total,        0 used,        0 free, 13316432 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
 8350 mpl       20   0 4534m 707m 5628 S   9.6  4.4   4:43.69 camput
 5901 mpl       20   0 1048m 279m 7964 S   0.3  1.7   0:55.94 firefox
 2521 mpl       20   0 1358m 104m  15m S   0.0  0.7   1:24.03 compiz
 8263 mpl       20   0  825m  77m 8808 S  31.9  0.5   6:14.29 camlistored
 1307 root      20   0  234m  67m  17m S   0.0  0.4   0:53.79 Xorg
 2922 mpl       30  10  663m  59m 1248 S   0.0  0.4   0:07.77 update-manager

so yeah, it looks like the goroutines were piling up because of rmdir taking 
too long, good catch.
The new (attached) pprof does not even show rmdir I think.

Interestingly, I still reach almost 100% of the mem being used.

Also it looks like camput is slowly leaking as well. towards the end it was:
 8350 mpl       20   0 4717m 898m 5628 S   4.3  5.6   6:33.77 camput
But that's less concerning.

Original comment by mathieu....@gmail.com on 11 Nov 2013 at 6:52

Attachments:

• pprof8703.0.svg

[GoogleCodeExporter]

since queuecreator is out, maybe this issue is gone. I'll wait for things to 
stabilize so I can retest that.

Original comment by mathieu....@gmail.com on 25 Nov 2013 at 3:55

[GoogleCodeExporter]

retried same test on 31a6685f1d76562aa28149bc1fe835f5352e3f93, seems ok now.

Original comment by mathieu....@gmail.com on 2 Dec 2013 at 12:22

• Changed state: Fixed

[GoogleCodeExporter]

This issue has moved to https://camlistore.org/issue/250

Original comment by bradfitz on 14 Dec 2014 at 11:36

• Added labels: IssueMoved, Restrict-AddIssueComment-Commit