Opportunity for performance improvement: require less RAM for PCA

[thanneken]

While doing some processing at the workshop I stumbled on an opportunity for improvement in the RAM requirements for PCA. It's worth verifying, but with the same data on the same machine I had "out of memory" fails when running PCA on 16-bit files. It did NOT fail on 32-bit files. I know that is counter intuitive, but 32-bit is floating point, and 16-bit is on a scale from 0-65,535. I think the math is easier with floating point. If this is correct, it should be a simple matter to set Image Type to 32-bit just before running PCA.

[thehabes]

I can do this conversion and do a run analysis comparison to see if there is improvement. This is a good thing to try, it may be a cheap improvement.

[thanneken]

I used top to track RAM use through a run with PCA. Unfortunately I don't have a baseline for comparison, but I did notice something interesting. I don't know if "garbage collection" is an issue in ImageJ2 and java plugins the way it was in ImageJ and macros. I did notice that the %MEM didn't go down when it should have. Even after the plugin stopped running it stayed at 41.6% until I clicked the status bar (which manually invokes garbage collection). When PCA was running it was only 33% so that should have been the peak, more or less. [PCA ran on 16 slices of 110 MB each on a machine with 32 GB RAM]

[thehabes]

http://forum.imagej.net/t/memory-not-clearing-over-time/2137 https://stackoverflow.com/questions/22912063/automatically-release-unused-memory-in-imagej-fiji

It is a known issue that ImageJ (whatever version you have installed) does not release memory like it should. Every time you run another thing, more memory is allocated to whatever ImageJ software is running and not released once the task completes. I can usually only do two or three RTI tasks before I have to restart ImageJ because I get the "Out Of Memory" message.

I run System.gc() and IJ.run("Collect Garbage"); at points in the plugin and every time the plugin completes, but it doesn't do anything. I have not used a JavaVM to trace the RAM usage of the plugin yet and have not try to change my OS garbage collector. It is possible we have to import some sort of custom java class that's a garbage collector so we can pass a fix with the plugin, but I have not investigated this yet.

There are real deep dives we can take into this and track what parts of the plugin use the most RAM (although I suspect PCA), but tracking this around could take a long time and lead to nothing. If you read those threads and decide it is something you want to pursue, we can talk about it.

[thanneken]

I suspected it would be too good to be true that a few lines of garbage collection would have made for easy improvement. For now I would think of this as belonging to the category of things that should work but don't. Let's leave those lines in even if they don't help. Maybe a future update of ImageJ or Java will improve the situation.