Reduce memory consumption of daemons and client

[turicas]

Some users have reported that the daemons are consuming too much memory, even when there are no jobs (memory is not freed).

I think the problem is worse in Broker and PipelineManager, but we don't know yet what is causing this. Need to do some benchmarks and optimizations in the code.

Related to #39.

[turicas]

Some links that could help:

• http://stackoverflow.com/questions/1316767/how-can-i-explicitly-free-memory-in-python/1316799#1316799
• http://docs.python.org/2/library/gc.html#gc.garbage
• http://stackoverflow.com/questions/9617001/python-garbage-collection-fails/9617534#9617534
• http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm
• http://pymotw.com/2/sys/limits.html

[turicas]

Probably worker processes on Broker need to be "refreshed" (killed and then re-started again) after execution of X jobs. As Python does not free memory for the operating system regularly when you destroy objects, the best way to do it is to kill the process.

Some time ago, all jobs were executed in fresh worker processes (see https://github.com/NAMD/pypln.backend/commit/19aa104a38f0b2365a3a14441887a5efa27284ff), then we shifted to this new approach: long-running worker process (they start when Broker starts and are killed when Broker is killed). Maybe we need a solution in the middle of these two.

[turicas]

These projects may help:

• memory_profiler
• pytracemalloc

[fccoelho]

@turicas , See this, straight from the multiprocessing documentation:

New in version 2.7: maxtasksperchild is the number of tasks a worker process can complete before it will exit and be replaced with a fresh worker process, to enable unused resources to be freed. The default maxtasksperchild is None, which means worker processes will live as long as the pool.

[turicas]

@fccoelho, thanks! Currently I'm not using multiprocessing.Pool (I've created my own Pool class) but I'll read the documentation carefully to decide if it's better to change to it or not.