search

OpenGATE / Gate

Official public repository of Gate
http://www.opengatecollaboration.org
GNU Lesser General Public License v3.0
236 stars 263 forks source link

Report of GATEv8.2 Bugs for Dosimetry simulation using voxelized phantoms (memory leak) #305

Open konhat88 opened 4 years ago

konhat88 commented 4 years ago

I have observe a memory leak (RAM) when executing dosimetry simulations. I am sharing an example. I have not include the real moby phantom, just a random image with its characteristics. If you run the simulation you will observe that slowly uses more and more RAM memory.

github.zip

djboersma commented 4 years ago

@konhat88 thanks for the example. I am running 3 different tests:

  1. Run your macro with Gate v8.2 release build: linked with geant4 10.05.p01: indeed, the RAM usage grows on my machine with whopping 3-4 megabyte per second. After 1h20min of running it has ballooned to 11.7 GiB.
  2. Run your macro with Gate v8.2 debug build, again linked with geant4 10.05.p01, under valgrind with smaller number of primaries. This is still running (with debug and valgrind it gets extremely slow). In an initial trial run with only a few thousand primaries valgrind reported a lot of sometimes boring, sometimes fascinating errors (which we should look into), but for the dose actor I only see that the memory for the dose files (& related images) is apparently not freed at the end (maybe the actors do not get deleted). That is an issue, but this leak does not scale with the number of primaries, so that is not the leak we are looking for.
  3. Run your macro with the latest "develop" version, linked with geant4 10.06.p01. Here the leak behavior is almost non-existent. During the first minute of running the RAM use of the GATE process grew to 664 megabyte ("RES" column in top or htop), and 17 minutes later it is at 665 megabyte. I would love to conclude that apparently the memory leak has been fixed by some commit after the 8.2 release, but I'm actually not so sure. In the GATE stdout/stderr stream there is a HUGE number G4 exceptions that seem to be caused by a small geometry error. Using /geometry/test/run I see indeed two overlaps of 5 Angstrom, which could be rounding errors. Which should be irrelevant, but from the exceptions it looks like none of tracks is getting anywhere, so the dose actor maybe never gets called and does not get any opportunity to leak.

Somehow this problem seems similar to issue https://github.com/OpenGATE/Gate/issues/257 , only with the KillActor instead of the DoseActor. The culprit code in that issue was in the SD code (see PR #266 by @cpommranz ), which would explain why I do not see such a big memory leak in the dose actor in my proton therapy simulations.

To be continued...

djboersma commented 4 years ago

I reran test number 1 with the line /gate/moby_phan/attachPhantomSD commented out. Now there does not seem to be any memory leak at all. This seems to confirm my suspicion that this problem is similar to the KillActor issue: when an actor with step action (like KillActor, DoseActor) is attached to a volume that is also as an "SD" attached to it, then you get a memory leak in Gate v8.2 and earlier. It would be nice if someone who is savvy about the internals of the Gate imaging code (I'm a radiotherapy guy) could confirm this.

konhat88 commented 4 years ago

Thank you, for your time. I hope for the best with Gate_V9

djboersma commented 4 years ago

I think this issue is effectively solved. Please re-open if you disagree. :)

albertnew2012 commented 3 years ago

I would like to share my finding here of memory leak involving the use of Dose actor in GATE v8.2, in case it could help anyone in the future. On top of commenting out /gate/***/attachPhantomSD, the following code still causes memory leak:

#=====================================================
#   START
#=====================================================
/gate/application/setTimeSlice   0.001 s
/gate/application/setTimeStart   60 s
/gate/application/setTimeStop    120 s
/gate/application/startDAQ

The slice here is 0.001s, the RAM usage increase quickly, I guess for each time slice GATE creates a frame of the geometry, but it doesn't release it when it moves to the next time slice, which is how RAM usage is built up. One way to circumvent this issue is to use a larger time step.

albertnew2012 commented 2 years ago

Not sure why this discussion was closed anyway, but the memory leakage issue caused by voxelized phantom has never been solved.

dsarrut commented 2 years ago

Hi @ZhengzhiLiu, please have a look at the latest 9.2 version to see if this bug is still there. thanks David