Out of memory problem

[jmmroldan]

I am running a macro to individually process some images in a folder with a loop. For each iteration, it's a rather simple image preparation and then find circles with the Hough Circles plugin. The images I am running the plugin on are 1241x1754. There are approx. 100 circles in each image. I realized that the calls to this plugin are asynchronous, so I clean the results table at the beginning of each iteration and I wait for the results before proceeding with the next iteration. Also, I close every open image at the end of each iteration. However, after processing some images (fewer than 30), I get an out of memory error. I am aware of the way the Java Virtual Machine works with memory. I made a test opening a large number of images to see how the memory gets used by the virtual machine until my memory limit. If I close them, the memory is not released, but I can open a large number of images again. Also if I force the garbage collector to run, the system gets back this memory. This is not happening when I use the Hough Circle plugin. The memory is getting used, but then it's "blocked" ImageJ cannot use it anymore and if I force the garbage collector it's not getting back to the system. The only way of getting back this memory is closing ImageJ.

[Llamero]

ImageJ has a weird way of dealing with garbage collection. The guidelines strongly suggest to not call garbage collection within a plugin, and instead to allow ImageJ to call it when needed. Since a Hough circle transform is a 3D expansion of a 2D image, it will take as memory as image size (max radius - min radius) radius step. Therefore, the transform itself can take a fair amount of memory. That said, once the transform is complete, and it generates the outputs, the data is cleared. It is then up to ImageJ to call garbage collection. We've found that on some computers this process works fine, and we can process many large images with no memory leaks (even movies GBs in size), but on other computers they have a memory leak. I'm not sure what the difference is. You can call garbage collection within a macro using the command "call("java.lang.System.gc");" or "run("Collect Garbage");"

Hope this helps, and let me know if it works.

Cheers, Ben Smith

[Llamero]

This discussion should also help: http://forum.imagej.net/t/how-to-find-memory-leaks-in-plugins-what-needs-to-get-disposed/10211/4

[jmmroldan]

Thanks for the link. I will check if I can locate the memory leak. As I said, forcing garbage collection at the end of each iteration (with either of the commands you mentioned) is not working for the hough circle plugin, although it works fine for other macros without the plugin. Do you know if this memory leak that happens in some computers is related to the java version or the specific java virtual machine?

[Llamero]

In the link I sent, there is another link to a stack overflow discussion that expands on the problem: https://stackoverflow.com/questions/1481178/how-to-force-garbage-collection-in-java

Basically, to call garbage collection does not run it at that moment, but rather sends a non-deterministic request (similar to an if-else statement). As a result, the virtual machine still may not call garbage collector in time. My guess on what makes it stable on some platforms is simply the amount of RAM. For example, my workstation has 64GB of RAM, so it is unlikely to be filled before GC is called, but an 8GB system is likely to run into RAM issues. No one said JVM was perfect.

[khoroshyy]

Unfortunately, I have to say I am running in into the same issue. After processing a few pictures using a macro, 8Gb of my ram are getting filled up. I will have to try garbage collector yet.

[jmmroldan]

@khoroshyy Could you share what Java Virtual Machine are you using? Did you have luck with forcing garbage collection? I ended up moving my whole solution to python/opencv. It's much faster and I am not running into any memory issues, but I would like to make it work in ImageJ too because it can be useful for other projects.

[imagejan]

Please note that there's also an ImageJ-Ops implementation of Hough Circle Transform being discussed here: https://github.com/imagej/imagej-ops/pull/526

@jmmroldan it would be awesome if you could test this and report on that PR whether it works for you, and how it compares performance-wise.

[khoroshyy]

@jmmroldan Hi. Sorry for late reply. I use Java 1,8,0_172 ImageJ 1.52 64 bit on Ubuntu 16.04. Garbage collection did not help. I am not sure what I have been using on my windows system as I deleted windows from my computer already. Python would be my choice of language but I am tied up to ImageJ by requirements in our lab .....

[Llamero]

I can say that the math for a basic Hough Circle Transform isn't too complex, and could even be implemented as a macro, where you draw a circle for each point in a binarized mask, and then sum the circles together. As such, it wouldn't be too hard to write an implementation specific to your needs, especially in languages like R, Python, or Matlab, where then you would simply search the binarized image for non-zero pixels, and then draw Bresenham circles, centered on each pixel, of the corresponding radius in a new transform matrix. This can be multi-threaded by dividing the range of radii you want to search by the number of processors, and then have each processor transform a subset of the radii. Same thing with the search for circles in the transform space, where each processor can search a block of that space, and then compare the results to find the highest scoring circle.

For the implementation I wrote, the key extra components I wanted to include are a fairly tuneable interface to adapt the search space to your sample, multi-threading to speed up both the transform and the search for circles, as well as the local search to greatly speed up circle tracking (a trick our lab needed, and seemed like other people would benefit from).

As far as the memory leak goes, all I know is that it seems to be platform dependent, because my particular platform has 64 GB of RAM, and we transform movies that are about 3-5GB in size, and the RAM stabilizes after the first movie with no memory leak (we process movies via a macro in blocks of up to 11 movies). This leads me to believe, along with a quick web search, that the problem is a Java garbage collection problem, where as well as I can understand it, the RAM used for the transform should be released once the plugin completes, like what we see on our computer. However, just because it is flagged for release does not mean that Java will immediately clear up that block for re-use (i.e. garbage collection is non-deterministic). That said, if there is a way to force a release of the RAM after the transform completes, I would definitely like to include it (and I'm certain that a lot of other people who write plugins would benefit from this code), but I couldn't find such a solution online, and instead only found references to how Java's garbage collection is non-deterministic, and therefore cannot be forced.

A quick web search also shows that OpenCV has an implementation of the Hough Circle transform, but I haven't tried it, so I'm not sure how tuneable it is, or if it is multi-threaded or if it has the same memory leak problem. If anyone tries it, I'd definitely like to know how it compares.

Hope this helps, Ben Smith

[esiefker]

This is a really severe bug. This simple macro increases memory use about 400mb each time I run it. I need radii from several dozen images, which consumes more memory than I have.

newImage("Untitled", "8-bit white", 1024, 1024, 1); setForegroundColor(0, 0, 0); makeOval(90, 90, 852, 852); run("Fill", "slice"); run("Convert to Mask"); run("Find Edges"); run("Hough Circle Transform","minRadius=400, maxRadius=500, inc=1, minCircles=1, maxCircles=65535, threshold=0.5, resolution=1000, ratio=1.0, bandwidth=10, local_radius=10, reduce results_table"); wait(60000); run("Close All"); run("Collect Garbage");"

[imagejan]

See this forum topic where alternatives are discussed.

[Llamero]

Could you send me the data you are working with. I think I may have a solution I would like to try, but our own implementations don't show the memory leak.

Basically, I'm not sure how FIJI handles plugins, but my random guess would be if FIJI is calling a new instance of the plugin each time it is called in a macro without clearing the previous instance, then you'll have a bunch of instances stack up, causing a memory leak. A partial solution would be to null out the large arrays used to construct the transform and outputs, so that the final heap footprint of the plugin will be much smaller once it finished running. It's a stab in the dark, but easy enough to implement, that it is worth a try.

[esiefker]

I'm home for the weekend, but I included a reproducible example in my post. I just tested it again on my home linux box (work is Windows 7) and got the same behavior.

[Llamero]

It turns out my hunch was correct. The leak was being caused by FIJI not releasing the instance of the plugin, and therefore each time it was called, it was creating a new instance while saving the prior instance. To fix this, I simply added a function to null out all of the transform arrays, which reduced the memory load by the 4th run, and so I then also added gc(), which reduced the memory load every run with no loss in speed. The new version has been uploaded to FIJI, so just run the updater to get the new version.

[Llamero]

Also here are my modifications to the test code. The most important revision that should help you is rather than using a fixed time delay, I have the macro wait until there are new results. This way, it continues right after the plugin is finished, no matter how long the plugin takes.

for(a=0; a<10; a++){
    newImage("Untitled", "8-bit white", 1024, 1024, 1);
    setForegroundColor(0, 0, 0);
    makeOval(90, 90, 852, 852);
    run("Fill", "slice");
    run("Convert to Mask");
    run("Find Edges");
    results = nResults;
    run("Hough Circle Transform","minRadius=400, maxRadius=500, inc=1, minCircles=1, maxCircles=65535, threshold=0.5, resolution=1000, ratio=1.0, bandwidth=10, local_radius=10, reduce results_table");
    while(results == nResults);
    run("Close All");
    memory = IJ.freeMemory();
    memory = replace(memory, "MB.*", "");
    print(memory);
}

[Llamero]

Attached is the result of running the above code: