Stop memory consumption increasing linearly with time

rahmans1 commented 2 months ago

Is your feature request related to a problem? Please describe. For a wide range of datasets in simulation production eicrecon shows linear increase in memory consumption over time. The plot below demonstrates an example of this issues by comparing the memory consumption over time in eicrecon vs npsim for one of these datasets.

pythia8CCDIS.pdf

Describe the solution you'd like We would prefer the rss to be under 2 GB in eicrecon for the duration of the run. This will allow us to achieve better throughput in Open Science Grid resources. The first step would be to identify why the memory consumption increases linearly without any obvious signs of memory leak.

Describe alternatives you've considered

Additional context

wdconinc commented 2 months ago

If we instrument the eicrecon-dis job here with prmon, does it show the same behavior? I.e. can we set something up that tracks this and flags degradation?

wdconinc commented 2 months ago

Entirely anecdotal, but I don't notice this on my own system which has a few differences from the containers we're running here, notably jana2@master and podio@0.99.

veprbl commented 2 months ago

I'm not able to reproduce with my local setup either (JANA2 2.2.1-rc1, podio 0.99).

    GB
1.896^                                     ##                                 
     |                   ::@:  ::::::::::::# ::@@:::::::::::::::::::  :::@::: 
     |          ::::@:@@:: @::::: : : : :: # : @ :: ::: :: ::: :: ::::: :@: ::
     |          :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |       ::::: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |       :: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |      ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |      ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |      ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |      ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |      ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |    @@::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |    @ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |    @ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |   @@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |   @@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |   @@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |   @@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |::@@@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
     |: @@@ ::: :: :@:@ :: @:: :: : : : :: # : @ :: ::: :: ::: :: ::: : :@: ::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   635.0

@rahmans1 Can you run with valgrind --tool=massif --massif-out-file=massif.out?

veprbl commented 2 months ago

That was for single particle ^^

I ran a long job over pythia8CCDIS_10x100_minQ2=1000_beamEffects_xAngle=-0.025_hiDiv_1.hepmc3.tree.root

    GB
1.914^                                                                       :
     |      @@@ :::::::  :::::@@@@:@@@@:::::::::::@@::::::::::::::::#:::::::@@
     |  ::::@@ :: :: : :::: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
     | :: : @@ :: :: : : :: ::@@ @:@ @ : ::: : : :@ ::: :::: :: : ::#: :::::@@
   0 +----------------------------------------------------------------------->Ti
     0                                                                   9.501

Can't see a leak, still.

veprbl commented 2 months ago

Re-running with npsim in the 24.05.0-stable container, I'm able to reproduce:

    GB
2.399^                                                                      : 
     |                                                          @@@@@@:@@:##::
     |                                           @@@@@@@@@@::@::@ @ @ :@ :# ::
     |                                   ::::@@:@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |                        ::::@@::@@@::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |           ::@@:::::::::::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |     ::::::: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |  :::: :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# ::
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |  :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |:::: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |: :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |: :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |: :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
     |: :: : :: :: @ :: :: :: ::: @@: @ @::: @ :@@ @@@ @ @ ::@: @ @ @ :@ :# :@
   0 +----------------------------------------------------------------------->Ti
     0                                                                   3.226

ms_print massif.out | grep IrtCherenkovParticleID

| | ->04.21% (82,426,288B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->04.21% (82,426,288B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->09.26% (196,608,608B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->09.26% (196,608,608B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->09.81% (210,377,160B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->09.81% (210,377,160B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->10.69% (231,974,064B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->10.69% (231,974,064B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->11.17% (244,414,800B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->11.17% (244,414,800B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->12.47% (281,128,512B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->12.47% (281,128,512B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->13.27% (305,147,528B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->13.27% (305,147,528B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->13.54% (314,657,336B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->13.54% (314,657,336B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->13.99% (325,824,296B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->13.99% (325,824,296B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->14.33% (336,300,216B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->14.33% (336,300,216B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->14.59% (343,495,320B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->14.59% (343,495,320B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->15.12% (358,452,752B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->15.12% (358,452,752B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->15.67% (375,047,008B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->15.67% (375,047,008B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->16.70% (407,772,904B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->16.70% (407,772,904B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->17.37% (428,545,040B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->17.37% (428,545,040B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->17.77% (441,215,152B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->17.77% (441,215,152B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->18.05% (447,386,632B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->18.05% (447,386,632B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->18.26% (459,564,168B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->18.26% (459,564,168B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->18.97% (482,290,312B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->18.97% (482,290,312B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->19.79% (508,054,456B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->19.79% (508,054,456B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)
| | ->43.17% (532,159,880B) 0x2D6EDFC7: eicrecon::IrtCherenkovParticleID::process(std::tuple<gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::TrackSegmentCollection const*>, gsl::not_null<edm4eic::RawTrackerHitCollection const*>, gsl::not_null<edm4eic::MCRecoTrackerHitAssociationCollection const*> > const&, std::tuple<gsl::not_null<edm4eic::CherenkovParticleIDCollection*>, gsl::not_null<edm4eic::CherenkovParticleIDCollection*> > const&) const (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/libalgorithms_pid.so)
| |   ->43.17% (532,159,880B) 0x5FB7AFD5: JOmniFactory<eicrecon::IrtCherenkovParticleID_factory, eicrecon::IrtCherenkovParticleIDConfig>::Process(std::shared_ptr<JEvent const> const&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.13.2-h4vo6zmtjaus2iiemtifefuh2ozz3pxe/lib/EICrecon/plugins/DRICH.so)

The upper frame looks like:

->46.58% (574,210,296B) 0x4B9A578: TStorage::ObjectAlloc(unsigned long) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/root-6.30.02-lzyvoi75d5zywtptko3ptcj42t22a5pp/lib/root/libCore.so.6.30.02)
| ->43.17% (532,159,880B) 0x96664A7: ChargedParticle::PIDReconstruction(CherenkovPID&) (in /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/irt-1.0.7-t5b4ycnb3lxj6yb543ck6qtbj5mzubsu/lib/libIRT.so)

veprbl commented 2 months ago

Reproducer:

mc cp S3/eictest/EPIC/EVGEN/DIS/CC/10x100/minQ2=1000/pythia8CCDIS_10x100_minQ2=1000_beamEffects_xAngle=-0.025_hiDiv_1.hepmc3.tree.root .
source /opt/detector/epic-24.05.0/bin/thisepic.sh
npsim --runType batch --filter.tracker edep0 --numberOfEvents 1000 --compactFile "$DETECTOR_PATH/$DETECTOR_CONFIG.xml" --inputFile pythia8CCDIS_10x100_minQ2\=1000_beamEffects_xAngle\=-0.025_hiDiv_1.hepmc3.tree.root --outputFile sim.edm4hep.root 
varlgrind --tool=massif --massif-out-file=massif.out eicrecon sim.edm4hep.root
ms_print massif.out

c-dilks commented 2 months ago

libIRT is leaky: https://github.com/eic/irt/issues/36

IIRC, some deletes are needed, but they may need to be within #ifdef DISABLE_ROOT_IO blocks, since ROOT may also delete things.

eic / EICrecon

Stop memory consumption increasing linearly with time #1456