[ROOT6_X] SIGSEGV in Validation/Geometry/materialBudget*Plots tests in G4PhysicalVolumeStore::Clean()

[iarspider]

In ROOT6_X IB, two unit tests in Validation/Geometry failed:

===== Test "materialBudgetHGCalPlots" ====
+ geom=Extended2026D92
+ VGEO_DIR=/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_ROOT6_X_2024-02-18-2300/src/Validation/Geometry/test
+ TEST_DIR=.
+ cmsRun /data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_ROOT6_X_2024-02-18-2300/src/Validation/Geometry/test/single_neutrino_cfg.py nEvents=1
+ python_cmd=python2
+ python3 -c 'from FWCore.PythonFramework.CmsRun import CmsRun'
+ python_cmd=python3
+ python3 /data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_ROOT6_X_2024-02-18-2300/src/Validation/Geometry/test/runP_HGCal_cfg.py geom=Extended2026D92 label=HGCal
/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_ROOT6_X_2024-02-18-2300/src/Validation/Geometry/test/genHGCalPlots.sh: line 11: 2433712 Segmentation fault      (core dumped) ${python_cmd} ${VGEO_DIR}/runP_HGCal_cfg.py geom=${geom} label=HGCal > $TEST_DIR/runP_HGCal_cfg_${geom}.log 2>&1

---> test materialBudgetHGCalPlots had ERRORS
TestTime:180
^^^^ End Test materialBudgetHGCalPlots ^^^^

===== Test "materialBudgetTrackerPlots" ====
+ VGEO_DIR=/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_ROOT6_X_2024-02-18-2300/src/Validation/Geometry/test
+ TEST_DIR=.
+ cmsRun /data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_ROOT6_X_2024-02-18-2300/src/Validation/Geometry/test/single_neutrino_cfg.py nEvents=1000
+ python_cmd=python2
+ python3 -c 'from FWCore.PythonFramework.CmsRun import CmsRun'
+ python_cmd=python3
+ for geom in {'Extended2015','Extended2017Plan1'}
+ python3 /data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_ROOT6_X_2024-02-18-2300/src/Validation/Geometry/test/runP_Tracker.py geom=Extended2015 label=Tracker
/data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_14_1_ROOT6_X_2024-02-18-2300/src/Validation/Geometry/test/genTrackerPlots.sh: line 11: 2433833 Segmentation fault      (core dumped) ${python_cmd} ${VGEO_DIR}/runP_Tracker.py geom=$geom label=Tracker > $TEST_DIR/runP_Tracker_cfg.log 2>&1

---> test materialBudgetTrackerPlots had ERRORS
TestTime:87
^^^^ End Test materialBudgetTrackerPlots ^^^^
>> Tests for package Validation/Geometry ran.

By running the test locally, I have extracted the actual crash log:

materialBudgetTrackerPlots

``` A fatal system signal has occurred: segmentation violation The following is the call stack containing the origin of the signal. Mon Feb 19 11:58:49 CET 2024 Thread 4 (Thread 0x7efdca55f700 (LWP 22302) "python3"): #0 0x00007efe07c0945c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007efde08efbdb in omt::ThreadHandoff::threadLoop (iArgs=0xc0421a0) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/condition_variable:102 #2 0x00007efe07c031ca in start_thread () from /lib64/libpthread.so.0 #3 0x00007efe070e5e73 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7efdcad60700 (LWP 22299) "python3"): #0 0x00007efe07c0945c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007efde08ed30b in operator() (__closure=0xaf6bef8) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/condition_variable:102 #2 0x00007efe052b5a73 in std::execute_native_thread_routine (__p=0xaf6bef0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #3 0x00007efe07c031ca in start_thread () from /lib64/libpthread.so.0 #4 0x00007efe070e5e73 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7efde7785700 (LWP 22297) "python3"): #0 0x00007efe07c0d672 in waitpid () from /lib64/libpthread.so.0 #1 0x00007efde92f4e17 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-02-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #2 0x00007efde92f7cea in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-02-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #3 0x00007efe052b5a73 in std::execute_native_thread_routine (__p=0x8b77f80) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007efe07c031ca in start_thread () from /lib64/libpthread.so.0 #5 0x00007efe070e5e73 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7efe08615f00 (LWP 22252) "python3"): #0 0x00007efe071d0301 in poll () from /lib64/libc.so.6 #1 0x00007efde93436af in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-02-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #2 0x00007efde92f7dbc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-02-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #3 0x00007efde92f8720 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-02-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #4 #5 0x00007efde0cb2a83 in G4PVPlacement::__dt_base () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_amd64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_amd64_gcc12/external/geant4/11.1.2-64a948d0da50afc1b46609c8e2c0e393/geant4.11.1.2/source/geometry/management/src/G4VPhysicalVolume.cc:167 #6 G4PVPlacement::__dt_del () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_amd64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_amd64_gcc12/external/geant4/11.1.2-64a948d0da50afc1b46609c8e2c0e393/geant4.11.1.2/source/geometry/volumes/src/G4PVPlacement.cc:159 #7 0x00007efde0d51588 in G4PhysicalVolumeStore::Clean() [clone .part.0] () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_amd64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_amd64_gcc12/external/geant4/11.1.2-64a948d0da50afc1b46609c8e2c0e393/geant4.11.1.2/source/geometry/management/src/G4PhysicalVolumeStore.cc:98 #8 0x00007efde0d51695 in G4PhysicalVolumeStore::~G4PhysicalVolumeStore () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_amd64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_amd64_gcc12/external/geant4/11.1.2-64a948d0da50afc1b46609c8e2c0e393/geant4.11.1.2/source/geometry/management/src/G4PhysicalVolumeStore.cc:80 #9 0x00007efe070fd26c in __run_exit_handlers () from /lib64/libc.so.6 #10 0x00007efe070fd3a0 in exit () from /lib64/libc.so.6 #11 0x00007efe070e6d8c in __libc_start_main () from /lib64/libc.so.6 #12 0x000000000040107e in _start () Current Modules: Module: none (crashed) ```

materialBudgetHGCalPlots

``` A fatal system signal has occurred: segmentation violation The following is the call stack containing the origin of the signal. Mon Feb 19 12:00:35 CET 2024 Thread 4 (Thread 0x7f804783c700 (LWP 22301) "python3"): #0 0x00007f808559345c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f805dbbdbdb in omt::ThreadHandoff::threadLoop (iArgs=0x2f81270) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/condition_variable:102 #2 0x00007f808558d1ca in start_thread () from /lib64/libpthread.so.0 #3 0x00007f8084a6fe73 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f804803d700 (LWP 22300) "python3"): #0 0x00007f808559345c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f805dbbb30b in operator() (__closure=0xb5fee78) at /data/cmsbld/jenkins/workspace/build-any-ib/w/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/condition_variable:102 #2 0x00007f8082c3fa73 in std::execute_native_thread_routine (__p=0xb5fee70) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #3 0x00007f808558d1ca in start_thread () from /lib64/libpthread.so.0 #4 0x00007f8084a6fe73 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f806507e700 (LWP 22296) "python3"): #0 0x00007f8085597672 in waitpid () from /lib64/libpthread.so.0 #1 0x00007f8066bf7e17 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-02-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #2 0x00007f8066bfacea in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-02-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #3 0x00007f8082c3fa73 in std::execute_native_thread_routine (__p=0x8e44120) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82 #4 0x00007f808558d1ca in start_thread () from /lib64/libpthread.so.0 #5 0x00007f8084a6fe73 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f8085f9df00 (LWP 22251) "python3"): #0 0x00007f8084b5a301 in poll () from /lib64/libc.so.6 #1 0x00007f8066c466af in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-02-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #2 0x00007f8066bfadbc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-02-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #3 0x00007f8066bfb720 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_ROOT6_X_2024-02-18-2300/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so #4 #5 0x00007f805df80a83 in G4PVPlacement::__dt_base () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_amd64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_amd64_gcc12/external/geant4/11.1.2-64a948d0da50afc1b46609c8e2c0e393/geant4.11.1.2/source/geometry/management/src/G4VPhysicalVolume.cc:167 #6 G4PVPlacement::__dt_del () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_amd64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_amd64_gcc12/external/geant4/11.1.2-64a948d0da50afc1b46609c8e2c0e393/geant4.11.1.2/source/geometry/volumes/src/G4PVPlacement.cc:159 #7 0x00007f805e01f588 in G4PhysicalVolumeStore::Clean() [clone .part.0] () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_amd64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_amd64_gcc12/external/geant4/11.1.2-64a948d0da50afc1b46609c8e2c0e393/geant4.11.1.2/source/geometry/management/src/G4PhysicalVolumeStore.cc:98 #8 0x00007f805e01f695 in G4PhysicalVolumeStore::~G4PhysicalVolumeStore () at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre3-el8_amd64_gcc12/build/CMSSW_14_0_0_pre3-build/BUILD/el8_amd64_gcc12/external/geant4/11.1.2-64a948d0da50afc1b46609c8e2c0e393/geant4.11.1.2/source/geometry/management/src/G4PhysicalVolumeStore.cc:80 #9 0x00007f8084a8726c in __run_exit_handlers () from /lib64/libc.so.6 #10 0x00007f8084a873a0 in exit () from /lib64/libc.so.6 #11 0x00007f8084a70d8c in __libc_start_main () from /lib64/libc.so.6 #12 0x000000000040107e in _start () Current Modules: Module: none (crashed) A fatal system signal has occurred: segmentation violation ```

[iarspider]

assign Validation/Geometry

[cmsbuild]

New categories assigned: dqm,geometry

@Dr15Jones,@civanch,@bsunanda,@makortel,@mdhildreth,@rvenditti,@syuvivida,@tjavaid,@nothingface0,@antoniovagnerini you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @iarspider.

@Dr15Jones, @makortel, @sextonkennedy, @antoniovilela, @smuzaffar, @rappoccio can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

The ROOT was updated for this IB (https://github.com/cms-sw/cmsdist/pull/8999)

On a cursory look all the CMSSW PRs merged in that IB look unrelated

[civanch]

@iarspider , can you, please, confirm if the problem is fixed when #44068 was merged or not yet?

[iarspider]

Yes, it is fixed. Thanks @civanch !