search

cBio / cbio-cluster

MSKCC cBio cluster documentation
12 stars 2 forks source link

Memory use Python Interpreter vs. OS #340

Closed rafwiewiora closed 8 years ago

rafwiewiora commented 9 years ago

So an issue presumably with the speed that Python interpreter returns memory to the OS:

I run a script that uses Python's multiprocessing module and runs 32 threads, with no shared memory. The script iterates over a database of around 110,000 files. memory_profiler shows around 50 MiB / thread memory use throughout the script run, so around 1.7 GB for all threads. However, I need to request a minimum of 15-18 GB of memory from qsub for the job not to be killed for resources violation.

The way I understand this from doing some reading, it looks like there's a 'lag' between the garbage collection done by the interpreter and the return of the memory to the OS. Is this right? How could I look into this in more detail? Is there anything I can do to speed up the memory return within my code?

Obviously it's not a big problem to request that 18 GB of memory if I'm using all the CPUs on one node, but I'm just curious as to why this happens and what would be a good way to predict how much memory I need to request in the future for other similar scripts without trialling different values and seeing what doesn't get killed.

Thanks!

jchodera commented 9 years ago

Does it help any if you do explicit cleanup in Python by using "del" on objects after you are done using them? That may reduce your reliance on the python gc. On Nov 18, 2015 6:04 PM, "Rafal Wiewiora" notifications@github.com wrote:

So an issue presumably with the speed that Python interpreter returns memory to the OS:

I run a script that uses Python's multiprocessing module and runs 32 threads, with no shared memory. The script iterates over a database of around 110,000 files. memory_profiler shows around 50 MiB / thread memory use throughout the script run, so around 1.7 GB for all threads. However, I need to request a minimum of 15-18 GB of memory from qsub for the job not to be killed for resources violation.

The way I understand this from doing some reading, it looks like there's a 'lag' between the garbage collection done by the interpreter and the return of the memory to the OS. Is this right? How could I look into this in more detail? Is there anything I can do to speed up the memory return within my code?

Obviously it's not a big problem to request that 18 GB of memory if I'm using all the CPUs on one node, but I'm just curious as to why this happens and what would be a good way to predict how much memory I need to request in the future for other similar scripts without trialling different values and seeing what doesn't get killed.

Thanks!

— Reply to this email directly or view it on GitHub https://github.com/cBio/cbio-cluster/issues/340.

tatarsky commented 9 years ago

Dim dim memory: isn't there a python setting to control the rate of garbage collection? Or a way to force a run? I will review old emails on this topic but I believe there is or I'm thinking of something else.

tatarsky commented 9 years ago

Is this relevant? https://pymotw.com/2/gc/

tatarsky commented 9 years ago

"Bump" as they say. Is this resolved?

rafwiewiora commented 9 years ago

@tatarsky: sorry have been swamped with other things, let me do more analysis on this this week.

tatarsky commented 9 years ago

No worries, just curious. Marking as held until said time.

rafwiewiora commented 8 years ago

Ok, resolved through finding my own mistake. Looks like the memory_profiler is not believable with multiprocessing and I was underestimating the memory use per thread by 6 times - the garbage collection looks very efficient in fact! Sorry for false alarm, good to close.

tatarsky commented 8 years ago

I am glad the GC is taking out the trash. Matter closed.