(Virtual) Memory explosion

ocaml-opam / Camelus

Bot posting reports on opam-repository pull-request using a web hook

Other

18 stars 9 forks source link

(Virtual) Memory explosion #1

Closed AltGr closed 6 years ago

AltGr commented 9 years ago

After a while, the program seems to fail on the server with "Cannot allocate memory" errors ; I couldn't reproduce locally, but have some results.

ocp-memprof didn't return any unexpected memory usage or leak ; memory peaks at 70MB when we load the git with all opam files in memory, which seems reasonable.
I traced system memory usage: Resident memory (VmRSS) stays very reasonable, however, virtual memory (VmSize) jumps to above 100GB... I guess that's where we hit a wall. This is using Linux's /proc/pid/status. @samoht, a clue what's happening ? I am guessing maybe too many mmaps ? Or am I doing something plainly wrong with git or lwt ? The peaks are reached during the opam_files function.

edwintorok commented 9 years ago

How many system threads do you have when this happens? AFAIK Lwt uses a pool of workers for blocking (i.e. non-socket) IO and the default for that is 1000 system threads that are created on demand, and each thread needs its own stack which is fairly large by default. I set a smaller value in my own program and that seems to reduce virtual memory usage: Lwt_unix.set_pool_size 64 Alternatively you could try reducing the stack size for the workers using pthread_attr_setstacksize

AltGr commented 9 years ago

@edwintorok thanks for the pointer. In these case it seems to stay at 2 threads though, which makes sense since only the web server and processing run in parallel (I wanted to keep things simple to begin with, and the processing writes to the git directory).

I've disabled the instability checks for now, as it's better to have just the lint than nothing after the bot has choked.

AltGr commented 9 years ago

I tried getting through the Git.Memory backend to see what happened: it fails at Git_memory.write_pack.

@samoht any hints ? Should I report to ocaml-git ?

AltGr commented 9 years ago

(mem otherwise jumps to ~900MB, and virtual just above 1GB, which is what I would have expected)

AltGr commented 9 years ago

I suspect the pack handling from ocaml-git; however, trying with ~unpack:true, I can't find the references anymore (i.e. nothing works)

shinzui commented 6 years ago

Is this still an open issue?

AltGr commented 6 years ago

Indeed, this has been fixed. I don't remember exactly which it was, but there were both fixes for ocaml-git, and an important fix for dose's installability check. Now it seems that after a month or so, there are too many threads left alive, but it's not really blocking and would be on lwt's side. Thanks!