Memory leak

[zihlmann]

There is still one or more memory leaks that are not so apparent when analyzing only one file. however if one analyzes many files at once one observes the resident memory to increase with time. As an example running over many files (>20) doing full tracking then one observes the resident memory increasing reaching more than 17GB after about 20 files analyzed. I should mention that this was seen while running on ifarm1402 (centos7)

some additional info running hd_root on ifarm1401 with no plugins with the following command:

hd_ana --nthreads=6 -PTHREAD_TIMEOUT=100 -PJANA:MAX_RELAUNCH_THREADS=10 /cache/halld/RunPeriod-2016-02/rawdata/Run010781/hd_rawdata_010781_000.evio /cache/halld/RunPeriod-2016-02/rawdata/Run010781/hd_rawdata_010781_001.evio /cache/halld/RunPeriod-2016-02/rawdata/Run010781/hd_rawdata_010781_002.evio /cache/halld/RunPeriod-2016-02/rawdata/Run010781/hd_rawdata_010781_003.evio /cache/halld/RunPeriod-2016-02/rawdata/Run010781/hd_rawdata_010781_004.evio /cache/halld/RunPeriod-2016-02/rawdata/Run010781/hd_rawdata_010781_005.evio /cache/halld/RunPeriod-2016-02/rawdata/Run010781/hd_rawdata_010781_006.evio /cache/halld/RunPeriod-2016-02/rawdata/Run010781/hd_rawdata_010781_007.evio

after the start and the first file open the resident memory is 170mb after opening of second file RES is 2.1Gb after opening of third file RES is 3.7Gb , halfway during this file 4.5Gb after opening of fourth file RES is 5.5Gb after opening of fifth file RES is 7.5Gb after opening of sixth file RES is 8.9Gb

[faustus123]

I’ve thought about this a little more and have a hypothesis on what could be causing this. The new parser employs lots of object pools to optimize performance. Each new file will have its own JEventSource_EVIOpp object that keeps it own pools. These objects are never deleted (technical reason that makes this difficult to do). I will take a look at making it delete these pools which may significantly reduce the “leak”.

Regards, -David

On Oct 28, 2016, at 8:39 AM, zihlmann notifications@github.com wrote:

There is still one or more memory leaks that are not so apparent when analyzing only one file. however if one analyzes many files at once one observes the resident memory to increase with time. As an example running over many files (>20) doing full tracking then one observes the resident memory increasing reaching more than 17GB after about 20 files analyzed.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/JeffersonLab/sim-recon/issues/610, or mute the thread https://github.com/notifications/unsubscribe-auth/AMPm5WFk1GAin2ypVkHsfgjM0EKHwCArks5q4e0XgaJpZM4KjZWm.

[zihlmann]

hi David, see: https://github.com/JeffersonLab/sim-recon/issues/610

running hd_root without any plugin and the leak is there. So I think your assessment is correct!

cheers, Beni

On 10/28/2016 01:28 PM, David Lawrence wrote:

I’ve thought about this a little more and have a hypothesis on what could be causing this. The new parser employs lots of object pools to optimize performance. Each new file will have its own JEventSource_EVIOpp object that keeps it own pools. These objects are never deleted (technical reason that makes this difficult to do). I will take a look at making it delete these pools which may significantly reduce the “leak”.

Regards, -David

On Oct 28, 2016, at 8:39 AM, zihlmann notifications@github.com wrote:

There is still one or more memory leaks that are not so apparent when analyzing only one file. however if one analyzes many files at once one observes the resident memory to increase with time. As an example running over many files (>20) doing full tracking then one observes the resident memory increasing reaching more than 17GB after about 20 files analyzed.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/JeffersonLab/sim-recon/issues/610, or mute the thread https://github.com/notifications/unsubscribe-auth/AMPm5WFk1GAin2ypVkHsfgjM0EKHwCArks5q4e0XgaJpZM4KjZWm.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/JeffersonLab/sim-recon/issues/610#issuecomment-256979844, or mute the thread https://github.com/notifications/unsubscribe-auth/AMvwC-QMFhuLPYWFT6-J_cG-MuzT10ryks5q4jDBgaJpZM4KjZWm.

[amfoda]

Is there an option we can include in hd_root command to avoid the new parser temporarily ?

[faustus123]

The old parser is accessible via:

-PEVENT_SOURCE_TYPE=JEventSourceGenerator_EVIO

However, it does not understand the new f250 data format so will likely choke.

Regards, -David

On Nov 8, 2016, at 4:00 PM, amfodajlab notifications@github.com wrote:

Is there an option we can include in hd_root command to avoid the new parser temporarily ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/JeffersonLab/sim-recon/issues/610#issuecomment-259258084, or mute the thread https://github.com/notifications/unsubscribe-auth/AMPm5Y9yS3fL08BVSq3FDY5LAmoR3gh5ks5q8OLngaJpZM4KjZWm.

[markito3]

@zihlmann and @faustus123 , is this issue still an issue?

[faustus123]

I do not believe this is still an issue.

On Mar 24, 2017, at 11:55 AM, Mark M. Ito notifications@github.com wrote:

@zihlmann https://github.com/zihlmann and @faustus123 https://github.com/faustus123 , is this issue still an issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JeffersonLab/sim-recon/issues/610#issuecomment-289063096, or mute the thread https://github.com/notifications/unsubscribe-auth/AMPm5cu6Y7HiQvu5fPcQ5IPJH_yEtJDoks5ro-dUgaJpZM4KjZWm.

[markito3]

Prematurely closed. Mea culpa.

[mashephe]

Last week I fixed I memory leak that appeared when running over EVIO skim files. Perhaps this fixes this issue?