search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.32k forks source link

[PowerPC] SIGSEGV in ~SiPixelRawToDigi in relval 141.112 #43429

Open iarspider opened 11 months ago

iarspider commented 11 months ago

RelVal 141.112 step 3 crashed with SIGSEGV in SiPixelRawToDigi::~SiPixelRawToDigi:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Wed Nov 29 10:22:04 CET 2023
Thread 8 (Thread 0x1008b5819210 (LWP 86080) "cmsRun"):
#0  0x0000100002c9274c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000010000293a8dc in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/gcc-12.3.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/powerpc64le-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/gcc-12.3.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x0000100845e1e8c0 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#5  0x0000100845e1f480 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#6  0x0000100845e1f6b8 in std::_Function_handler<void (), Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#7  0x0000100845e1afe0 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#8  0x000010085518d2e8 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_framework.so.2
#9  0x0000100002c89718 in start_thread () from /lib64/libpthread.so.0
#10 0x0000100002e0b778 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x1008b5409210 (LWP 86079) "cmsRun"):
#0  0x0000100002c9274c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000010000293a8dc in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/gcc-12.3.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/powerpc64le-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/gcc-12.3.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x0000100845e1e8c0 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#5  0x0000100845e1f480 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#6  0x0000100845e1f6b8 in std::_Function_handler<void (), Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#7  0x0000100845e1afe0 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#8  0x000010085518d2e8 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_framework.so.2
#9  0x0000100002c89718 in start_thread () from /lib64/libpthread.so.0
#10 0x0000100002e0b778 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x1008b2dd9210 (LWP 86078) "cmsRun"):
#0  0x0000100002c9274c in pthread_cond_wait@@GLIBC_2.17 () from /lib64/libpthread.so.0
#1  0x000010000293a8dc in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/gcc-12.3.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/powerpc64le-redhat-linux-gnu/bits/gthr-default.h:865
#2  std::__condvar::wait (__m=..., this=<optimized out>) at /scratch/cmsbuild/jenkins_a/workspace/jenkins-test-bootstrap/bootstrap/BUILD/el8_ppc64le_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/gcc-12.3.1/obj/powerpc64le-redhat-linux-gnu/libstdc++-v3/include/bits/std_mutex.h:155
#3  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:41
#4  0x0000100845e1e8c0 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WaitForWork(Eigen::EventCount::Waiter*, tsl::thread::EigenEnvironment::Task*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#5  0x0000100845e1f480 in Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#6  0x0000100845e1f6b8 in std::_Function_handler<void (), Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#7  0x0000100845e1afe0 in std::_Function_handler<void (), tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_cc.so.2
#8  0x000010085518d2e8 in tsl::(anonymous namespace)::PThread::ThreadFn(void*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/external/el8_ppc64le_gcc12/lib/libtensorflow_framework.so.2
#9  0x0000100002c89718 in start_thread () from /lib64/libpthread.so.0
#10 0x0000100002e0b778 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x100867a09210 (LWP 85704) "cmsRun"):
#0  0x0000100002dc5b04 in nanosleep () from /lib64/libc.so.6
#1  0x0000100002dc5940 in sleep () from /lib64/libc.so.6
#2  0x0000100009a2447c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000100002e02ac4 in syscall () from /lib64/libc.so.6
#5  0x000010000259e308 in tbb::detail::r1::futex_wait (comparand=2, futex=0x100008c10024) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x100008c10024) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x100008c10020) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x100008c10000) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x100008c10000) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000100002c89718 in start_thread () from /lib64/libpthread.so.0
#11 0x0000100002e0b778 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x100863c99210 (LWP 85703) "cmsRun"):
#0  0x0000100002dc5b04 in nanosleep () from /lib64/libc.so.6
#1  0x0000100002dc5940 in sleep () from /lib64/libc.so.6
#2  0x0000100009a2447c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000100002e02ac4 in syscall () from /lib64/libc.so.6
#5  0x000010000259e308 in tbb::detail::r1::futex_wait (comparand=2, futex=0x100008c10124) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x100008c10124) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x100008c10120) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x100008c10100) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x100008c10100) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000100002c89718 in start_thread () from /lib64/libpthread.so.0
#11 0x0000100002e0b778 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x100863289210 (LWP 85702) "cmsRun"):
#0  0x0000100002dc5b04 in nanosleep () from /lib64/libc.so.6
#1  0x0000100002dc5940 in sleep () from /lib64/libc.so.6
#2  0x0000100009a2447c in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000100002e02ac4 in syscall () from /lib64/libc.so.6
#5  0x000010000259e308 in tbb::detail::r1::futex_wait (comparand=2, futex=0x100008c100a4) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/semaphore.h:100
#6  tbb::detail::r1::binary_semaphore::P (this=0x100008c100a4) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/semaphore.h:253
#7  tbb::detail::r1::rml::internal::thread_monitor::wait (this=0x100008c100a0) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/rml_thread_monitor.h:235
#8  tbb::detail::r1::rml::private_worker::run (this=0x100008c10080) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/private_server.cpp:273
#9  tbb::detail::r1::rml::private_worker::thread_routine (arg=0x100008c10080) at /scratch/cmsbuild/jenkins_a/workspace/auto-builds/CMSSW_13_3_0_pre4-el8_ppc64le_gcc12/build/CMSSW_13_3_0_pre4-build/BUILD/el8_ppc64le_gcc12/external/tbb/v2021.9.0-b095b6d0e5df24a3b5f9bdd21e523ee3/tbb-v2021.9.0/src/tbb/private_server.cpp:221
#10 0x0000100002c89718 in start_thread () from /lib64/libpthread.so.0
#11 0x0000100002e0b778 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x100831fa9210 (LWP 85578) "cmsRun"):
#0  0x0000100002c99510 in waitpid () from /lib64/libpthread.so.0
#1  0x0000100009a2475c in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000100009a2657c in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000100009a2b414 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)()> > >::_M_run() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginFWCoreServicesPlugins.so
#4  0x0000100002944fa0 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#5  0x0000100002c89718 in start_thread () from /lib64/libpthread.so.0
#6  0x0000100002e0b778 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x1000027acf70 (LWP 82420) "cmsRun"):
#0  0x0000100002df9e70 in poll () from /lib64/libc.so.6
#1  0x0000100009a24ec4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000100009a26658 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000100009a2a12c in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  edata_arena_ind_get (edata=0x0) at include/jemalloc/internal/edata.h:258
#6  tcache_bin_flush_match (small=true, cur_binshard=<optimized out>, cur_arena_ind=2, edata=0x0) at src/tcache.c:301
#7  tcache_bin_flush_impl (small=true, nflush=118, ptrs=0x3fffed875de0, binind=<optimized out>, cache_bin=<optimized out>, tcache=<optimized out>, tsd=0x1000027b2b90) at src/tcache.c:434
#8  tcache_bin_flush_bottom (small=true, rem=<optimized out>, binind=<optimized out>, cache_bin=<optimized out>, tcache=<optimized out>, tsd=0x1000027b2b90) at src/tcache.c:519
#9  je_tcache_bin_flush_small (tsd=0x1000027b2b90, tcache=<optimized out>, cache_bin=<optimized out>, binind=<optimized out>, rem=<optimized out>) at src/tcache.c:529
#10 0x0000100001f9202c in tcache_gc_small (tsd=<optimized out>, tcache_slow=<optimized out>, tcache=<optimized out>, szind=<optimized out>) at src/tcache.c:148
#11 0x0000100001f941dc in je_tcache_gc_dalloc_event_handler () at src/tcache.c:223
#12 0x0000100001f9751c in je_te_event_trigger (tsd=0x1000027b2b90, ctx=<optimized out>) at src/thread_event.c:299
#13 0x0000100001f296f4 in te_event_advance (is_alloc=<optimized out>, usize=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/thread_event.h:287
#14 thread_dalloc_event (usize=<optimized out>, tsd=<optimized out>) at include/jemalloc/internal/thread_event.h:293
#15 isfree (slow_path=<optimized out>, tcache=<optimized out>, usize=<optimized out>, ptr=<optimized out>, tsd=<optimized out>) at src/jemalloc.c:2991
#16 je_sdallocx_default (ptr=<optimized out>, size=<optimized out>, flags=<optimized out>) at src/jemalloc.c:3928
#17 0x0000100001f99430 in sizedDeleteImpl (size=<optimized out>, ptr=<optimized out>) at src/jemalloc_cpp.cpp:195
#18 operator delete (ptr=<optimized out>, size=<optimized out>) at src/jemalloc_cpp.cpp:200
#19 0x000010087c3f23b8 in SiPixelRawToDigi::~SiPixelRawToDigi() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginEventFilterSiPixelRawToDigiPlugins.so
#20 0x000010087c3f26b0 in SiPixelRawToDigi::~SiPixelRawToDigi() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginEventFilterSiPixelRawToDigiPlugins.so
#21 0x000010000044a7b4 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::deleteModulesEarly() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/libFWCoreFramework.so
#22 0x000010087c3f7eac in edm::stream::ProducingModuleAdaptor<SiPixelRawToDigi, edm::stream::EDProducerBase, edm::stream::EDProducerAdaptorBase>::~ProducingModuleAdaptor() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginEventFilterSiPixelRawToDigiPlugins.so
#23 0x000010083ad89f40 in std::_Sp_counted_deleter<edm::stream::EDProducerAdaptorBase*, std::default_delete<edm::stream::EDProducerAdaptorBase>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginRecoBTagCombinedPlugins.so
#24 0x000010083ad8e104 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginRecoBTagCombinedPlugins.so
#25 0x000010083ad8ed2c in std::_Sp_counted_ptr_inplace<edm::maker::ModuleHolderT<edm::stream::EDProducerAdaptorBase>, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/pluginRecoBTagCombinedPlugins.so
#26 0x00001000001ebc48 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/libFWCoreFramework.so
#27 0x000010000039b954 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/libFWCoreFramework.so
#28 0x000010000039b940 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/libFWCoreFramework.so
#29 0x000010000039b940 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/libFWCoreFramework.so
#30 0x000010000039b940 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/libFWCoreFramework.so
#31 0x000010000039b940 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > > >::_M_erase(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, edm::propagate_const<std::shared_ptr<edm::maker::ModuleHolder> > > >*) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/libFWCoreFramework.so
#32 0x000010000039c114 in std::_Sp_counted_ptr_inplace<edm::ModuleRegistry, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/libFWCoreFramework.so
#33 0x00001000001ebc48 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/libFWCoreFramework.so
#34 0x000010000025fc20 in edm::Schedule::~Schedule() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/libFWCoreFramework.so
#35 0x0000100000230c14 in edm::EventProcessor::~EventProcessor() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02813/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2023-11-28-2300/lib/el8_ppc64le_gcc12/libFWCoreFramework.so
#36 0x00000000100064e8 in (anonymous namespace)::EventProcessorWithSentry::~EventProcessorWithSentry() ()
#37 0x0000000010005b94 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: none

A fatal system signal has occurred: segmentation violation
cmsbuild commented 11 months ago

A new Issue was created by @iarspider .

@Dr15Jones, @sextonkennedy, @antoniovilela, @makortel, @smuzaffar, @rappoccio can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

iarspider commented 11 months ago

assign EventFilter/SiPixelRawToDigi

cmsbuild commented 11 months ago

New categories assigned: reconstruction

@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 11 months ago

The segfault is in jemalloc, so this could be a sign of incorrect memory management about anywhere (as jemalloc is quite picky on memory being managed correctly, and can crash somewhere if that is not the case).