search

LBEM-CH / focus

High Throughput Electron Microscopy Image Processing Software
http://www.focus-em.org
GNU General Public License v2.0
35 stars 14 forks source link

Memory leak in 2dx_merge #109

Closed sthennin closed 8 years ago

sthennin commented 9 years ago

2dx_merge on OS X can grow to >10 GB in RAM volume, and also is extremely slow in reaction.

When running scripts that produce results in the LOGS/${scriptname}.results file, then 2dx_merge grows in size. This can be tested in the following way: Modify the "Custom Script 1" so that it outputs for every of the 500 images some 100 times the same line echo "set dummy = 1" >> LOGS/${scriptname}.results If you then launch 2dx_merge on this script, 2dx_merge explores in RAM memory. It is as if for every copy of a Parameter that it wants to save, it is reading an entire copy of the configuration file and then forgetting it in memory.

Other related issues: I operate 2dx_merge on a dataset with 550 images. Loading 2dx_merge takes about 15 minutes, even though parsing through 550 2dx_image.cfg files should only take one minute.

2dx_merge writes its own local 2dx_merge.cfg file many times per second. Is this needed?

nikhilbiyani commented 9 years ago

@sthennin : What's the update on this one? Should this be looked on?

sthennin commented 8 years ago

Is avoided by not writing out too many parameters into the RESULTS window, but rather updating the parameters via a Python script. Upgrade to an SQL database would eliminate this problem.