search

macronucleus / Chromagnon

Image correction software for chromatic shifts in fluorescence microscopic images
GNU General Public License v2.0
21 stars 5 forks source link

Memory leak? #22

Closed jstitlow closed 3 years ago

jstitlow commented 5 years ago

Hi Atsushi, Chromagnon is crashing out after processing ~15 images ~(3,50,512,512), both on W7 and Linux.

Running from GUI in W7 (Chromagnon v0.70) it shuts down and gives a generic error message.

On Linux server running from command line (Chromagnon 0.81 & 0.70) it just fills up the memory (125GB) and goes very slow.

Have you run into this before and do you have any idea what could be causing it?

Thanks! Josh

jstitlow commented 5 years ago

Looking at the W7 instance, Chromagnon has finished processing the data, but it is still using 9GB memory and the aligned images are unavailable (i.e., can't be moved out of their current directory).

jstitlow commented 5 years ago

Same with MacOS GUI (0.70), after finishing, Chromagnon is still holding several GBs of data in memory.

Are you able to modify the code so that it deletes the data after processing each image?

macronucleus commented 5 years ago

Hi Josh, No, I never had this problem, but I assume the file formats that I don't use could cause it. Chromagnon is supposed to remove all data after processing. Can I ask you the file format? If it is a 'tiff' format, let me know what kind of software the files are made from. --- Atsushi

jstitlow commented 5 years ago

File format is likely to be the issue.

The original 6-channel .ome.tiffs (from Volocity, Perkin Elmer) are split into two separate 3-channel .tif files using Image-J.

I then load ~100 reference images with their associated target image into Chromagnon GUI (or process them in a bash loop, or a Python loop calling subprocess).

Is there a preferred format for exporting images from image-J?

I will try to check if the behavior is the same for other file types or if I save the same files using tifffile.

jstitlow commented 5 years ago

The issue could be to do with holding all of the reference images in memory. Using the chromagnon.csv output files, Chromagnon can process the same 100 target images without memory problems.

So a solution is to process the files in 2 steps, first to generate the .csv files then to apply the correction.

macronucleus commented 5 years ago

I see. ImageJ format should be okay, so something is wrong. I will take a look. Thank you for finding the temporary solution.

macronucleus commented 5 years ago

I found large images are sometimes stored to speed up fitting. I disabled it so that it does not consume lots of memory. I believe the newer version (v0.82) does not end up with the memory problem.

jstitlow commented 5 years ago

Thanks! Will check this out at the end of next week. Ultimately I didn't observe the problem with the MacOS v0.81 distribution. The problem was specific to 0.81 cloned from GitHub running either on Linux (Debian) or MacOS. j

On Fri, Aug 2, 2019 at 12:27 AM macronucleus notifications@github.com wrote:

I found large images are sometimes stored to speed up fitting. I disabled it so that it does not consume lots of memory. I believe the newer version (v0.82) does not end up with the memory problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/macronucleus/Chromagnon/issues/22?email_source=notifications&email_token=ABMMCBSECM3AYSLXVMHCES3QCNWMRA5CNFSM4IFWNZ2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3ME75Q#issuecomment-517492726, or mute the thread https://github.com/notifications/unsubscribe-auth/ABMMCBT7LCR7KSFJ63JPOBDQCNWMRANCNFSM4IFWNZ2A .