search

c-cube / frog-utils

[frozen] Scheduling and running jobs on a shared computer, then analyse their output
BSD 2-Clause "Simplified" License
5 stars 0 forks source link

Froglock/frogmap seems to freeze above at about 1018 tasks #28

Closed Gbury closed 8 years ago

Gbury commented 8 years ago

While running some benchs lately, i have seen that when doing a lot of tests (i.e about 20000), frogmap seems to freeze after doing 1018 tasks, using internal locking (i.e --lock true option, which is the default). Additionally, when trying to check what happens with froglock status, froglock seemed to be frozen too.

This might be a very problematic issue.

Gbury commented 8 years ago

Seems that not using the shiny --lock=true, and instead doing something like frogmap --lock=false -- 'froglock cmd' solves the problem, so I think the cause is that file descriptors opened by frogmap are not cleaned/closed soon enough and the frogmap call reaches the file descriptors limit (which can be seen with ulimit -{S,H}n). So while there is a temporary fix, it might be nice to see how we could clean frogmap's sub processes file descriptors in time.

c-cube commented 8 years ago

I don't remember, but I thought most of IOs were made through with_foo functions…

Gbury commented 8 years ago

Closed with afc19e9e6220799dc73da632faabe35699f016be