search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.3k forks source link

[ROOT6 and ROOT628] Unit tests `Miscellanea` and `PrimaryVertex` failing #42468

Closed aandvalenzuela closed 1 year ago

aandvalenzuela commented 1 year ago

Hello,

Unit tests Miscellanea and PrimaryVertex (module Alignment/OfflineValidation) are failing in both ROOT6 and ROOT628 IBs with:

Executing rm StripHitMap.coor
tempfilename StripHitMap.coor
created palette with 1 colors
printing StripHitMap.png
Executing rm StripHitMap.coor

Fatal system signal has occurred during exit
/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_13_3_ROOT6_X_2023-08-03-2300/src/Alignment/OfflineValidation/test/testingScripts/test_unitMiscellanea.sh: line 5: 2017541 Aborted                 (core dumped) cmsRun ${CMSSW_BASE}/src/Alignment/OfflineValidation/test/inspectData_cfg.py unitTest=True trackCollection=ALCARECOTkAlCosmicsCTF0T
Failure running inspectData_cfg.py: status 134

at https://github.com/cms-sw/cmssw/blob/master/Alignment/OfflineValidation/test/testingScripts/test_unitMiscellanea.sh#L5 and

Severity    # Occurrences   Total Occurrences
--------    -------------   -----------------
Info                 3659                3659
FwkInfo               100                 100
Warning               431                 431
Error                   1                   1
System                  3                   3

dropped waiting message count 0

Fatal system signal has occurred during exit
/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_13_3_ROOT6_X_2023-08-03-2300/src/Alignment/OfflineValidation/test/testingScripts/test_unitPrimaryVertex.sh: line 6: 2017591 Aborted                 (core dumped) cmsRun ${CMSSW_BASE}/src/Alignment/OfflineValidation/test/testPrimaryVertexRelatedValidations_cfg.py isPhase2=False maxEvents=100
Failure running testPrimaryVertexRelatedValidations_cfg.py isPhase2=False: status 134

at https://github.com/cms-sw/cmssw/blob/master/Alignment/OfflineValidation/test/testingScripts/test_unitPrimaryVertex.sh#L6, respectively.

They started failing at the same time yesterday CMSSW_13_3_ROOT6_X_2023-08-02-2300. Today they failed again CMSSW_13_3_ROOT6_X_2023-08-03-2300. I can reproduce the crashes locally.

cmsbuild commented 1 year ago

A new Issue was created by @aandvalenzuela Andrea Valenzuela.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

aandvalenzuela commented 1 year ago

This issue seems related to the change to TCMalloc (https://github.com/cms-sw/cmssw/pull/42440) since the failing cmsRun command runs fine when using cmsRunJE instead (cmsRunJE Alignment/OfflineValidation/test/inspectData_cfg.py unitTest=True trackCollection=ALCARECOTkAlCosmicsCTF0T).

makortel commented 1 year ago

assign alca

cmsbuild commented 1 year ago

New categories assigned: alca

@perrotta,@consuegs,@francescobrivio,@saumyaphor4252,@tvami you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 1 year ago

@aandvalenzuela Would you be able to run the failing cmsRun job in gdb and print the stack trace of the point of crash?

aandvalenzuela commented 1 year ago

Sure, here it goes:

Executing rm StripHitMap.coor
[Detaching after vfork from child process 15381]

Thread 1 "cmsRun" received signal SIGSEGV, Segmentation fault.
tcmalloc::SLL_PopRange (end=<synthetic pointer>, start=<synthetic pointer>, N=32, head=0x45a0a0) at src/linked_list.h:88
88  src/linked_list.h: No such file or directory.
(gdb) 
(gdb) t a a bt

Thread 7 (Thread 0x7fff9a025700 (LWP 29262) "cmsRun"):
#0  0x00007ffff4ff5da6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007ffff4ff5e98 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007ffff2c763f6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08-03-2300/external/el8_amd64_gcc11/lib/libXrdCl.so.3
#3  0x00007ffff2c764a9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08-03-2300/external/el8_amd64_gcc11/lib/libXrdCl.so.3
#4  0x00007ffff4fed1ca in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff4c59e73 in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fff9a826700 (LWP 29261) "cmsRun"):
#0  0x00007ffff4ff5da6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007ffff4ff5e98 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007ffff2c763f6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08-03-2300/external/el8_amd64_gcc11/lib/libXrdCl.so.3
#3  0x00007ffff2c764a9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08-03-2300/external/el8_amd64_gcc11/lib/libXrdCl.so.3
#4  0x00007ffff4fed1ca in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff4c59e73 in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fff9b027700 (LWP 29260) "cmsRun"):
#0  0x00007ffff4ff5da6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x00007ffff4ff5e98 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x00007ffff2c763f6 in XrdCl::JobManager::RunJobs() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08-03-2300/external/el8_amd64_gcc11/lib/libXrdCl.so.3
#3  0x00007ffff2c764a9 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08-03-2300/external/el8_amd64_gcc11/lib/libXrdCl.so.3
#4  0x00007ffff4fed1ca in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff4c59e73 in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fff9b828700 (LWP 29259) "cmsRun"):
#0  0x00007ffff4ff7180 in nanosleep () from /lib64/libpthread.so.0
#1  0x00007ffff2d63328 in XrdSysTimer::Wait(int) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08-03-2300/external/el8_amd64_gcc11/lib/libXrdUtils.so.3
#2  0x00007ffff2be3509 in XrdCl::TaskManager::RunTasks() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08---Type <RET> for more, q to quit, c to continue without paging--
03-2300/external/el8_amd64_gcc11/lib/libXrdCl.so.3
#3  0x00007ffff2be3689 in RunRunnerThread () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08-03-2300/external/el8_amd64_gcc11/lib/libXrdCl.so.3
#4  0x00007ffff4fed1ca in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff4c59e73 in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fff9c029700 (LWP 29258) "cmsRun"):
#0  0x00007ffff4d4ee87 in epoll_wait () from /lib64/libc.so.6
#1  0x00007ffff2d5d6d2 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08-03-2300/external/el8_amd64_gcc11/lib/libXrdUtils.so.3
#2  0x00007ffff2d59e7d in XrdSys::IOEvents::BootStrap::Start(void*) () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08-03-2300/external/el8_amd64_gcc11/lib/libXrdUtils.so.3
#3  0x00007ffff2d629d8 in XrdSysThread_Xeq () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc11/cms/cmssw-patch/CMSSW_13_3_ROOT628_X_2023-08-03-2300/external/el8_amd64_gcc11/lib/libXrdUtils.so.3
#4  0x00007ffff4fed1ca in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff4c59e73 in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fffd4cb9700 (LWP 26564) "cmsRun"):
#0  0x00007ffff4ff6ab4 in read () from /lib64/libpthread.so.0
#1  0x00007ffff2f28caf in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-01-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#2  0x00007ffff2ef07df in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02796/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT628_X_2023-08-01-2300/lib/el8_amd64_gcc11/pluginFWCoreServicesPlugins.so
#3  0x00007ffff5678894 in std::execute_native_thread_routine (__p=0x761d520) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x00007ffff4fed1ca in start_thread () from /lib64/libpthread.so.0
#5  0x00007ffff4c59e73 in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffff4140c80 (LWP 24365) "cmsRun"):
#0  tcmalloc::SLL_PopRange (end=<synthetic pointer>, start=<synthetic pointer>, N=32, head=0x45a0a0) at src/linked_list.h:88
#1  tcmalloc::SLL_PopRange (end=<synthetic pointer>, start=<synthetic pointer>, N=32, head=0x45a0a0) at src/linked_list.h:79
#2  tcmalloc::ThreadCache::FreeList::PopRange (end=<synthetic pointer>, start=<synthetic pointer>, N=32, this=0x45a0a0) at src/thread_cache.h:238
#3  tcmalloc::ThreadCache::ReleaseToCentralCache (this=this@entry=0x45a040, src=src@entry=0x45a0a0, cl=<optimized out>, N=N@entry=32) at src/thread_cache.cc:206
#4  0x00007ffff57e0f2c in tcmalloc::ThreadCache::ListTooLong (this=0x45a040, list=0x45a0a0, cl=<optimized out>) at src/thread_cache.cc:164
#5  0x00007ffff6dbf521 in TString::UnLink (this=0xc93520) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.28.05-499b5524dd6fe178eb63581311edcc20/root-6.28.05/core/base/inc/TString.h:265
#6  0x00007ffff680de68 in TString::~TString (this=0xc93520, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/ro--Type <RET> for more, q to quit, c to continue without paging--
ot/6.28.05-499b5524dd6fe178eb63581311edcc20/root-6.28.05/core/base/src/TString.cxx:249
#7  0x00007ffff6817fd9 in TStyle::~TStyle (this=0xc93200, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.28.05-499b5524dd6fe178eb63581311edcc20/root-6.28.05/core/base/src/TStyle.cxx:478
#8  0x00007ffff68180c6 in TStyle::~TStyle (this=0xc93200, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.28.05-499b5524dd6fe178eb63581311edcc20/root-6.28.05/core/base/src/TStyle.cxx:483
#9  0x00007ffff686e092 in TCollection::GarbageCollect (obj=0xc93200) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.28.05-499b5524dd6fe178eb63581311edcc20/root-6.28.05/core/cont/src/TCollection.cxx:736
#10 0x00007ffff68765b3 in TList::Delete (this=0xc73c00, option=0x7ffff69d8064 "") at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.28.05-499b5524dd6fe178eb63581311edcc20/root-6.28.05/core/cont/src/TList.cxx:537
#11 0x00007ffff6798b76 in TROOT::EndOfProcessCleanups (this=0x7ffff6b17080 <ROOT::Internal::GetROOT1()::alloc>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.28.05-499b5524dd6fe178eb63581311edcc20/root-6.28.05/core/base/src/TROOT.cxx:1231
#12 0x00007ffff67a642e in CallEndOfProcessCleanups () at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.28.05-499b5524dd6fe178eb63581311edcc20/root-6.28.05/core/base/src/TApplication.cxx:90
#13 0x00007ffff4c7126c in __run_exit_handlers () from /lib64/libc.so.6
#14 0x00007ffff4c713a0 in exit () from /lib64/libc.so.6
#15 0x00007ffff4c5ad8c in __libc_start_main () from /lib64/libc.so.6
#16 0x000000000040803e in _start ()
makortel commented 1 year ago

Thanks! So the crash occurs inside TCMalloc, upon destruction of TString. Let's tag @pcanal (crash is visible in ROOT6 master and 6.28 IBs, but not in 6.26 production IBs)

dan131riley commented 1 year ago

I ran the PrimaryVertex test in valgrind memcheck, and while there were a bunch of mismatched new/free errors (which I'll investigate), it didn't reproduce the crash or have a clear smoking gun. Also doesn't reproduce with 1 thread. Reproduces readily with 4 threads, stack trace below--as with the one from @aandvalenzuela it's in TROOT::EndOfProcessCleanups(), but a different destructor. I'd guess there's a race condition somewhere in the cleanups, with a very large window. memcheck serializes, so as with running single-threaded it can suppress race conditions.

The only interesting part of the stack trace is thread 1:

Thread 1 (Thread 0x7f7fe717cc80 (LWP 78055) "cmsRun"):
#0  tcmalloc::SLL_PopRange (end=<synthetic pointer>, start=<synthetic pointer>, N=32, head=0x15180a0) at src/linked_list.h:88
#1  tcmalloc::SLL_PopRange (end=<synthetic pointer>, start=<synthetic pointer>, N=32, head=0x15180a0) at src/linked_list.h:79
#2  tcmalloc::ThreadCache::FreeList::PopRange (end=<synthetic pointer>, start=<synthetic pointer>, N=32, this=0x15180a0) at src/thread_cache.h:238
#3  tcmalloc::ThreadCache::ReleaseToCentralCache (this=this@entry=0x1518040, src=src@entry=0x15180a0, cl=cl@entry=3, N=592) at src/thread_cache.cc:201
#4  0x00007f7fe881cea6 in tcmalloc::ThreadCache::Scavenge (this=0x1518040) at src/thread_cache.cc:224
#5  0x00007f7fc4eb33d5 in TGeoStateInfo::~TGeoStateInfo (this=0x1655da00, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/geom/geom/src/TGeoStateInfo.cxx:70
#6  0x00007f7fc4e0ec8b in TGeoNodeCache::~TGeoNodeCache (this=0x2966a900, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/geom/geom/src/TGeoCache.cxx:114
#7  0x00007f7fc4e0ed4a in TGeoNodeCache::~TGeoNodeCache (this=0x2966a900, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/geom/geom/src/TGeoCache.cxx:119
#8  0x00007f7fc4e5d84d in TGeoNavigator::~TGeoNavigator (this=0x1d3758c0, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/geom/geom/src/TGeoNavigator.cxx:167
#9  0x00007f7fc4e5d8e8 in TGeoNavigator::~TGeoNavigator (this=0x1d3758c0, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/geom/geom/src/TGeoNavigator.cxx:170
#10 0x00007f7fe993374e in TCollection::GarbageCollect (obj=0x1d3758c0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/cont/src/TCollection.cxx:736
#11 0x00007f7fe99439e1 in TObjArray::Delete (this=0x2c635590) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/cont/src/TObjArray.cxx:376
#12 0x00007f7fe99428ae in TObjArray::~TObjArray (this=0x2c635590, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/cont/src/TObjArray.cxx:97
#13 0x00007f7fc4dd8174 in TGeoNavigatorArray::~TGeoNavigatorArray (this=0x2c635590, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/geom/geom/inc/TGeoNavigator.h:228
#14 0x00007f7fc4dd8190 in TGeoNavigatorArray::~TGeoNavigatorArray (this=0x2c635590, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/geom/geom/inc/TGeoNavigator.h:228
#15 0x00007f7fc4e3966d in TGeoManager::ClearNavigators (this=0x2a1bd400) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/geom/geom/src/TGeoManager.cxx:806
#16 0x00007f7fc4e386aa in TGeoManager::~TGeoManager (this=0x2a1bd400, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/geom/geom/src/TGeoManager.cxx:536
#17 0x00007f7fc4e38838 in TGeoManager::~TGeoManager (this=0x2a1bd400, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/geom/geom/src/TGeoManager.cxx:548
#18 0x00007f7fe993374e in TCollection::GarbageCollect (obj=0x2a1bd400) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/cont/src/TCollection.cxx:736
#19 0x00007f7fe993bc6f in TList::Delete (this=0x1d741c0, option=0x7f7fe9a28d24 "") at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/cont/src/TList.cxx:537
#20 0x00007f7fe98cf294 in TROOT::EndOfProcessCleanups (this=0x7f7fe9b6d040 <ROOT::Internal::GetROOT1()::alloc>) at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TROOT.cxx:1231
#21 0x00007f7fe9867f46 in CallEndOfProcessCleanups () at /data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TApplication.cxx:90
#22 0x00007f7fe7cad26c in __run_exit_handlers () from /lib64/libc.so.6
#23 0x00007f7fe7cad3a0 in exit () from /lib64/libc.so.6
#24 0x00007f7fe7c96d8c in __libc_start_main () from /lib64/libc.so.6
#25 0x000000000040803e in _start ()
dan131riley commented 1 year ago

I built a debug version of tcmalloc, which turns on a bunch of internal consistency checks. It was about as slow as valgrind(!), didn't reproduce the problem and didn't find any errors.

dan131riley commented 1 year ago

I believe the "mismatched new" warnings from valgrind are false positives, but I once I filtered those out I was left with 5 "Invalid free" errors similar to the ones in #40347:

==156325== Invalid free() / delete / delete[] / realloc()
==156325==    at 0x403CF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x5415285E: ???
==156325==    by 0x12AB5E16: cling::IncrementalExecutor::runAndRemoveStaticDestructors(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C0E8: cling::Interpreter::runAndRemoveStaticDestructors() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x128193B7: TCling::ResetGlobals() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:3709)
==156325==    by 0x60841D3: TROOT::EndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TROOT.cxx:1219)
==156325==    by 0x601CF45: CallEndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TApplication.cxx:90)
==156325==    by 0x788126B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==156325==    by 0x788139F: exit (in /usr/lib64/libc-2.28.so)
==156325==    by 0x786AD8B: (below main) (in /usr/lib64/libc-2.28.so)

==156325==  Address 0x39d158b0 is 0 bytes inside a block of size 29 free'd
==156325==    at 0x403CF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x5415285E: ???
==156325==    by 0x12AB5E16: cling::IncrementalExecutor::runAndRemoveStaticDestructors(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C0E8: cling::Interpreter::runAndRemoveStaticDestructors() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x128193B7: TCling::ResetGlobals() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:3709)
==156325==    by 0x60841D3: TROOT::EndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TROOT.cxx:1219)
==156325==    by 0x601CF45: CallEndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TApplication.cxx:90)
==156325==    by 0x788126B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==156325==    by 0x788139F: exit (in /usr/lib64/libc-2.28.so)
==156325==    by 0x786AD8B: (below main) (in /usr/lib64/libc-2.28.so)

==156325==  Block was alloc'd at
==156325==    at 0x403C0F3: operator new[](unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x54152AA1: ???
==156325==    by 0x541529C1: ???
==156325==    by 0x5415292F: ???
==156325==    by 0x541522B2: ???
==156325==    by 0x5415290B: ???
==156325==    by 0x5415291C: ???
==156325==    by 0x5415290F: ???
==156325==    by 0x14028FC1: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AB4DF5: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C7C7: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AC55C9: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AC8378: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A38A93: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12823C55: ExecAutoParse(char const*, bool, cling::Interpreter*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6304)
==156325==    by 0x128244DE: TCling::AutoParseImplRecurse(char const*, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6409)
==156325==    by 0x1282482D: TCling::AutoParseImplRecurse(char const*, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6445)
==156325==    by 0x12824B5E: TCling::AutoParse(char const*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6494)
==156325==    by 0x1280C444: TClingLookupHelper__AutoParse(char const*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:901)
==156325==    by 0x12666A4C: ROOT::TMetaUtils::TClingLookupHelper::GetPartiallyDesugaredNameWithScopeHandling(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/clingutils/src/TClingUtils.cxx:578)
==156325==    by 0x611C953: TClassEdit::TSplitType::ShortType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/foundation/src/TClassEdit.cxx:454)
==156325==    by 0x611DFEC: TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/foundation/src/TClassEdit.cxx:871)
==156325==    by 0x6140772: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/meta/src/TClass.cxx:3055)
==156325==    by 0x61403FC: TClass::GetClass(char const*, bool, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/meta/src/TClass.cxx:2970)
==156325==    by 0x4E3C798: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==156325==    by 0x4E3D014: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==156325==    by 0x4DB2DF5: edm::BranchDescription::initFromDictionary() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==156325==    by 0x4DB3230: edm::BranchDescription::BranchDescription(edm::BranchType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::Hash<1> const&, edm::TypeWithDict const&, bool, bool, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==156325==    by 0x4BF697D: edm::ProductRegistryHelper::addToRegistry(__gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, __gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, edm::ModuleDescription const&, edm::ProductRegistry&, edm::ProductRegistryHelper*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==156325==    by 0x4BF73B0: edm::ProducerBase::registerProducts(edm::ProducerBase*, edm::ProductRegistry*, edm::ModuleDescription const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)

==156325== Invalid free() / delete / delete[] / realloc()
==156325==    at 0x403CF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x5415286F: ???
==156325==    by 0x12AB5E16: cling::IncrementalExecutor::runAndRemoveStaticDestructors(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C0E8: cling::Interpreter::runAndRemoveStaticDestructors() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x128193B7: TCling::ResetGlobals() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:3709)
==156325==    by 0x60841D3: TROOT::EndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TROOT.cxx:1219)
==156325==    by 0x601CF45: CallEndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TApplication.cxx:90)
==156325==    by 0x788126B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==156325==    by 0x788139F: exit (in /usr/lib64/libc-2.28.so)
==156325==    by 0x786AD8B: (below main) (in /usr/lib64/libc-2.28.so)

==156325==  Address 0x33a7c0b0 is 0 bytes inside a block of size 18 free'd
==156325==    at 0x403CF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x5415286F: ???
==156325==    by 0x12AB5E16: cling::IncrementalExecutor::runAndRemoveStaticDestructors(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C0E8: cling::Interpreter::runAndRemoveStaticDestructors() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x128193B7: TCling::ResetGlobals() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:3709)
==156325==    by 0x60841D3: TROOT::EndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TROOT.cxx:1219)
==156325==    by 0x601CF45: CallEndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TApplication.cxx:90)
==156325==    by 0x788126B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==156325==    by 0x788139F: exit (in /usr/lib64/libc-2.28.so)
==156325==    by 0x786AD8B: (below main) (in /usr/lib64/libc-2.28.so)

==156325==  Block was alloc'd at
==156325==    at 0x403C0F3: operator new[](unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x54152AA1: ???
==156325==    by 0x541529C1: ???
==156325==    by 0x5415292F: ???
==156325==    by 0x54152278: ???
==156325==    by 0x5415290B: ???
==156325==    by 0x5415291C: ???
==156325==    by 0x5415290F: ???
==156325==    by 0x14028FC1: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AB4DF5: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C7C7: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AC55C9: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AC8378: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A38A93: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12823C55: ExecAutoParse(char const*, bool, cling::Interpreter*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6304)
==156325==    by 0x128244DE: TCling::AutoParseImplRecurse(char const*, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6409)
==156325==    by 0x1282482D: TCling::AutoParseImplRecurse(char const*, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6445)
==156325==    by 0x12824B5E: TCling::AutoParse(char const*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6494)
==156325==    by 0x1280C444: TClingLookupHelper__AutoParse(char const*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:901)
==156325==    by 0x12666A4C: ROOT::TMetaUtils::TClingLookupHelper::GetPartiallyDesugaredNameWithScopeHandling(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/clingutils/src/TClingUtils.cxx:578)
==156325==    by 0x611C953: TClassEdit::TSplitType::ShortType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/foundation/src/TClassEdit.cxx:454)
==156325==    by 0x611DFEC: TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/foundation/src/TClassEdit.cxx:871)
==156325==    by 0x6140772: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/meta/src/TClass.cxx:3055)
==156325==    by 0x61403FC: TClass::GetClass(char const*, bool, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/meta/src/TClass.cxx:2970)
==156325==    by 0x4E3C798: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==156325==    by 0x4E3D014: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==156325==    by 0x4DB2DF5: edm::BranchDescription::initFromDictionary() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==156325==    by 0x4DB3230: edm::BranchDescription::BranchDescription(edm::BranchType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::Hash<1> const&, edm::TypeWithDict const&, bool, bool, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==156325==    by 0x4BF697D: edm::ProductRegistryHelper::addToRegistry(__gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, __gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, edm::ModuleDescription const&, edm::ProductRegistry&, edm::ProductRegistryHelper*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==156325==    by 0x4BF73B0: edm::ProducerBase::registerProducts(edm::ProducerBase*, edm::ProductRegistry*, edm::ModuleDescription const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)

==156325== Invalid free() / delete / delete[] / realloc()
==156325==    at 0x403CF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x54152891: ???
==156325==    by 0x12AB5E16: cling::IncrementalExecutor::runAndRemoveStaticDestructors(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C0E8: cling::Interpreter::runAndRemoveStaticDestructors() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x128193B7: TCling::ResetGlobals() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:3709)
==156325==    by 0x60841D3: TROOT::EndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TROOT.cxx:1219)
==156325==    by 0x601CF45: CallEndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TApplication.cxx:90)
==156325==    by 0x788126B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==156325==    by 0x788139F: exit (in /usr/lib64/libc-2.28.so)
==156325==    by 0x786AD8B: (below main) (in /usr/lib64/libc-2.28.so)

==156325==  Address 0x3373bf20 is 0 bytes inside a block of size 20 free'd
==156325==    at 0x403CF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x54152891: ???
==156325==    by 0x12AB5E16: cling::IncrementalExecutor::runAndRemoveStaticDestructors(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C0E8: cling::Interpreter::runAndRemoveStaticDestructors() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x128193B7: TCling::ResetGlobals() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:3709)
==156325==    by 0x60841D3: TROOT::EndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TROOT.cxx:1219)
==156325==    by 0x601CF45: CallEndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TApplication.cxx:90)
==156325==    by 0x788126B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==156325==    by 0x788139F: exit (in /usr/lib64/libc-2.28.so)
==156325==    by 0x786AD8B: (below main) (in /usr/lib64/libc-2.28.so)

==156325==  Block was alloc'd at
==156325==    at 0x403C0F3: operator new[](unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x54152AA1: ???
==156325==    by 0x541529C1: ???
==156325==    by 0x5415292F: ???
==156325==    by 0x54152204: ???
==156325==    by 0x5415290B: ???
==156325==    by 0x5415291C: ???
==156325==    by 0x5415290F: ???
==156325==    by 0x14028FC1: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AB4DF5: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C7C7: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AC55C9: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AC8378: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A38A93: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12823C55: ExecAutoParse(char const*, bool, cling::Interpreter*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6304)
==156325==    by 0x128244DE: TCling::AutoParseImplRecurse(char const*, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6409)
==156325==    by 0x1282482D: TCling::AutoParseImplRecurse(char const*, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6445)
==156325==    by 0x12824B5E: TCling::AutoParse(char const*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6494)
==156325==    by 0x1280C444: TClingLookupHelper__AutoParse(char const*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:901)
==156325==    by 0x12666A4C: ROOT::TMetaUtils::TClingLookupHelper::GetPartiallyDesugaredNameWithScopeHandling(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/clingutils/src/TClingUtils.cxx:578)
==156325==    by 0x611C953: TClassEdit::TSplitType::ShortType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/foundation/src/TClassEdit.cxx:454)
==156325==    by 0x611DFEC: TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/foundation/src/TClassEdit.cxx:871)
==156325==    by 0x6140772: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/meta/src/TClass.cxx:3055)
==156325==    by 0x61403FC: TClass::GetClass(char const*, bool, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/meta/src/TClass.cxx:2970)
==156325==    by 0x4E3C798: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==156325==    by 0x4E3D014: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==156325==    by 0x4DB2DF5: edm::BranchDescription::initFromDictionary() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==156325==    by 0x4DB3230: edm::BranchDescription::BranchDescription(edm::BranchType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::Hash<1> const&, edm::TypeWithDict const&, bool, bool, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==156325==    by 0x4BF697D: edm::ProductRegistryHelper::addToRegistry(__gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, __gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, edm::ModuleDescription const&, edm::ProductRegistry&, edm::ProductRegistryHelper*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==156325==    by 0x4BF73B0: edm::ProducerBase::registerProducts(edm::ProducerBase*, edm::ProductRegistry*, edm::ModuleDescription const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)

==156325== Invalid free() / delete / delete[] / realloc()
==156325==    at 0x403CF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x541528B3: ???
==156325==    by 0x12AB5E16: cling::IncrementalExecutor::runAndRemoveStaticDestructors(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C0E8: cling::Interpreter::runAndRemoveStaticDestructors() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x128193B7: TCling::ResetGlobals() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:3709)
==156325==    by 0x60841D3: TROOT::EndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TROOT.cxx:1219)
==156325==    by 0x601CF45: CallEndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TApplication.cxx:90)
==156325==    by 0x788126B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==156325==    by 0x788139F: exit (in /usr/lib64/libc-2.28.so)
==156325==    by 0x786AD8B: (below main) (in /usr/lib64/libc-2.28.so)

==156325==  Address 0x32667e10 is 0 bytes inside a block of size 18 free'd
==156325==    at 0x403CF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x541528B3: ???
==156325==    by 0x12AB5E16: cling::IncrementalExecutor::runAndRemoveStaticDestructors(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C0E8: cling::Interpreter::runAndRemoveStaticDestructors() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x128193B7: TCling::ResetGlobals() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:3709)
==156325==    by 0x60841D3: TROOT::EndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TROOT.cxx:1219)
==156325==    by 0x601CF45: CallEndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TApplication.cxx:90)
==156325==    by 0x788126B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==156325==    by 0x788139F: exit (in /usr/lib64/libc-2.28.so)
==156325==    by 0x786AD8B: (below main) (in /usr/lib64/libc-2.28.so)

==156325==  Block was alloc'd at
==156325==    at 0x403C0F3: operator new[](unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x54152AA1: ???
==156325==    by 0x541529C1: ???
==156325==    by 0x5415292F: ???
==156325==    by 0x54152190: ???
==156325==    by 0x5415290B: ???
==156325==    by 0x5415291C: ???
==156325==    by 0x5415290F: ???
==156325==    by 0x14028FC1: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AB4DF5: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C7C7: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AC55C9: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AC8378: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A38A93: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12823C55: ExecAutoParse(char const*, bool, cling::Interpreter*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6304)
==156325==    by 0x128244DE: TCling::AutoParseImplRecurse(char const*, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6409)
==156325==    by 0x1282482D: TCling::AutoParseImplRecurse(char const*, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6445)
==156325==    by 0x12824B5E: TCling::AutoParse(char const*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6494)
==156325==    by 0x1280C444: TClingLookupHelper__AutoParse(char const*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:901)
==156325==    by 0x12666A4C: ROOT::TMetaUtils::TClingLookupHelper::GetPartiallyDesugaredNameWithScopeHandling(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/clingutils/src/TClingUtils.cxx:578)
==156325==    by 0x611C953: TClassEdit::TSplitType::ShortType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/foundation/src/TClassEdit.cxx:454)
==156325==    by 0x611DFEC: TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/foundation/src/TClassEdit.cxx:871)
==156325==    by 0x6140772: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/meta/src/TClass.cxx:3055)
==156325==    by 0x61403FC: TClass::GetClass(char const*, bool, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/meta/src/TClass.cxx:2970)
==156325==    by 0x4E3C798: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==156325==    by 0x4E3D014: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==156325==    by 0x4DB2DF5: edm::BranchDescription::initFromDictionary() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==156325==    by 0x4DB3230: edm::BranchDescription::BranchDescription(edm::BranchType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::Hash<1> const&, edm::TypeWithDict const&, bool, bool, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==156325==    by 0x4BF697D: edm::ProductRegistryHelper::addToRegistry(__gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, __gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, edm::ModuleDescription const&, edm::ProductRegistry&, edm::ProductRegistryHelper*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==156325==    by 0x4BF73B0: edm::ProducerBase::registerProducts(edm::ProducerBase*, edm::ProductRegistry*, edm::ModuleDescription const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)

==156325== Invalid free() / delete / delete[] / realloc()
==156325==    at 0x403CF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x541528C1: ???
==156325==    by 0x12AB5E16: cling::IncrementalExecutor::runAndRemoveStaticDestructors(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C0E8: cling::Interpreter::runAndRemoveStaticDestructors() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x128193B7: TCling::ResetGlobals() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:3709)
==156325==    by 0x60841D3: TROOT::EndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TROOT.cxx:1219)
==156325==    by 0x601CF45: CallEndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TApplication.cxx:90)
==156325==    by 0x788126B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==156325==    by 0x788139F: exit (in /usr/lib64/libc-2.28.so)
==156325==    by 0x786AD8B: (below main) (in /usr/lib64/libc-2.28.so)

==156325==  Address 0x325163a0 is 0 bytes inside a block of size 26 free'd
==156325==    at 0x403CF6C: free (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x541528C1: ???
==156325==    by 0x12AB5E16: cling::IncrementalExecutor::runAndRemoveStaticDestructors(cling::Transaction*) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C0E8: cling::Interpreter::runAndRemoveStaticDestructors() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x128193B7: TCling::ResetGlobals() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:3709)
==156325==    by 0x60841D3: TROOT::EndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TROOT.cxx:1219)
==156325==    by 0x601CF45: CallEndOfProcessCleanups() (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/base/src/TApplication.cxx:90)
==156325==    by 0x788126B: __run_exit_handlers (in /usr/lib64/libc-2.28.so)
==156325==    by 0x788139F: exit (in /usr/lib64/libc-2.28.so)
==156325==    by 0x786AD8B: (below main) (in /usr/lib64/libc-2.28.so)

==156325==  Block was alloc'd at
==156325==    at 0x403C0F3: operator new[](unsigned long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/external/valgrind/3.17.0-7bfcd2b5e4f162fb4b127c18285f46f6/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==156325==    by 0x54152AA1: ???
==156325==    by 0x541529C1: ???
==156325==    by 0x5415292F: ???
==156325==    by 0x54152156: ???
==156325==    by 0x5415290B: ???
==156325==    by 0x5415291C: ???
==156325==    by 0x5415290F: ???
==156325==    by 0x14028FC1: (anonymous namespace)::GenericLLVMIRPlatformSupport::initialize(llvm::orc::JITDylib&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AB4DF5: cling::IncrementalExecutor::runStaticInitializersOnce(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A3C7C7: cling::Interpreter::executeTransaction(cling::Transaction&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AC55C9: cling::IncrementalParser::commitTransaction(llvm::PointerIntPair<cling::Transaction*, 2u, cling::IncrementalParser::EParseResult, llvm::PointerLikeTypeTraits<cling::Transaction*>, llvm::PointerIntPairInfo<cling::Transaction*, 2u, llvm::PointerLikeTypeTraits<cling::Transaction*> > >&, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12AC8378: cling::IncrementalParser::Compile(llvm::StringRef, cling::CompilationOptions const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12A38A93: cling::Interpreter::parseForModule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/lib/libCling.so)
==156325==    by 0x12823C55: ExecAutoParse(char const*, bool, cling::Interpreter*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6304)
==156325==    by 0x128244DE: TCling::AutoParseImplRecurse(char const*, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6409)
==156325==    by 0x1282482D: TCling::AutoParseImplRecurse(char const*, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6445)
==156325==    by 0x12824B5E: TCling::AutoParse(char const*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:6494)
==156325==    by 0x1280C444: TClingLookupHelper__AutoParse(char const*) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/metacling/src/TCling.cxx:901)
==156325==    by 0x12666A4C: ROOT::TMetaUtils::TClingLookupHelper::GetPartiallyDesugaredNameWithScopeHandling(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/clingutils/src/TClingUtils.cxx:578)
==156325==    by 0x611C953: TClassEdit::TSplitType::ShortType(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, int) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/foundation/src/TClassEdit.cxx:454)
==156325==    by 0x611DFEC: TClassEdit::GetNormalizedName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::basic_string_view<char, std::char_traits<char> >) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/foundation/src/TClassEdit.cxx:871)
==156325==    by 0x6140772: TClass::GetClass(char const*, bool, bool, unsigned long, unsigned long) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/meta/src/TClass.cxx:3055)
==156325==    by 0x61403FC: TClass::GetClass(char const*, bool, bool) (/data/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/el8_amd64_gcc11/lcg/root/6.29.01-192f857c68d2f0a1a8cb821f03f5a854/root-6.29.01/core/meta/src/TClass.cxx:2970)
==156325==    by 0x4E3C798: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==156325==    by 0x4E3D014: edm::TypeWithDict::byName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so)
==156325==    by 0x4DB2DF5: edm::BranchDescription::initFromDictionary() (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==156325==    by 0x4DB3230: edm::BranchDescription::BranchDescription(edm::BranchType const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::Hash<1> const&, edm::TypeWithDict const&, bool, bool, std::set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libDataFormatsProvenance.so)
==156325==    by 0x4BF697D: edm::ProductRegistryHelper::addToRegistry(__gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, __gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, edm::ModuleDescription const&, edm::ProductRegistry&, edm::ProductRegistryHelper*, bool) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
==156325==    by 0x4BF73B0: edm::ProducerBase::registerProducts(edm::ProducerBase*, edm::ProductRegistry*, edm::ModuleDescription const&) (in /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02797/el8_amd64_gcc11/cms/cmssw/CMSSW_13_3_ROOT6_X_2023-08-07-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so)
makortel commented 1 year ago

assign core

cmsbuild commented 1 year ago

New categories assigned: core

@Dr15Jones,@smuzaffar,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 1 year ago

@pcanal @vgvassilev Could you take a look of the valgrind report in https://github.com/cms-sw/cmssw/issues/42468#issuecomment-1677987788 ?

mmusich commented 1 year ago

IIUC https://github.com/cms-sw/cmssw/pull/42628 solved this particular issue. Indeed in:

All unit tests consistently succeeded.

tvami commented 1 year ago

+alca

hahnjo commented 1 year ago

IIUC #42628 solved this particular issue.

Correct, #42628 works around the issue in CMSSW. Note that the general problem of const objects with non-trivial constructors and destructors is still pending on the ROOT side, I will resume working on it this week after a short break.

mmusich commented 1 year ago

Given that the real fix has to happen in the Core ROOT s/w and the IBs failures are solved I am wondering if this issue could be closed.

makortel commented 1 year ago

+core

makortel commented 1 year ago

@cmsbuild, please close

cmsbuild commented 1 year ago

This issue is fully signed and ready to be closed.