search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.07k stars 4.28k forks source link

[PPC64LE] SIGSEGV in PhysicsTools/SelectorUtils/testPhysicsToolsSelectorUtilsPythonTestsDriver #43688

Closed iarspider closed 2 weeks ago

iarspider commented 8 months ago

Test PhysicsTools/SelectorUtils/testPhysicsToolsSelectorUtilsPythonTestsDriver failed with SIGSEGV inside ROOT code:

===========================================================
There was a crash (#6 0x0000100001699e70 in SigHandler(ESignals) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libCore.so).
This is the entire stack trace of all threads:
===========================================================
#0  0x00001000008a5634 in waitpid () from /lib64/libc.so.6
#1  0x000010000080b3ac in do_system () from /lib64/libc.so.6
#2  0x00001000005d9d58 in system_compat () from /lib64/libpthread.so.0
#3  0x000010000169a59c in TUnixSystem::StackTrace() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libCore.so
#4  0x00001000011bf348 in (anonymous namespace)::TExceptionHandlerImp::HandleException(int) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libcppyy_backend3_9.so
#5  0x0000100001699bd8 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libCore.so
#6  0x0000100001699e70 in SigHandler(ESignals) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libCore.so
#7  0x000010000168eae8 in sighandler(int) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libCore.so
#8  <signal handler called>
#9  0x000010000473a9e0 in llvm::orc::ExecutionSession::lookup(std::vector<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags>, std::allocator<std::pair<llvm::orc::JITDylib*, llvm::orc::JITDylibLookupFlags> > > const&, llvm::orc::SymbolStringPtr, llvm::orc::SymbolState) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-09-2300/external/el8_ppc64le_gcc12/lib/libCling.so
#10 0x000010000475789c in llvm::orc::LLJIT::lookupLinkerMangled(llvm::orc::JITDylib&, llvm::orc::SymbolStringPtr) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-09-2300/external/el8_ppc64le_gcc12/lib/libCling.so
#11 0x0000100002f4bcec in cling::IncrementalJIT::getSymbolAddress(llvm::StringRef, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-09-2300/external/el8_ppc64le_gcc12/lib/libCling.so
#12 0x0000100002f43f18 in cling::IncrementalExecutor::getPointerToGlobalFromJIT(llvm::StringRef) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-09-2300/external/el8_ppc64le_gcc12/lib/libCling.so
#13 0x0000100002ea4834 in cling::Interpreter::compileFunction(llvm::StringRef, llvm::StringRef, bool, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-09-2300/external/el8_ppc64le_gcc12/lib/libCling.so
#14 0x0000100002dffe18 in TClingCallFunc::compile_wrapper(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-09-2300/external/el8_ppc64le_gcc12/lib/libCling.so
#15 0x0000100002e0ccc8 in TClingCallFunc::make_wrapper() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-09-2300/external/el8_ppc64le_gcc12/lib/libCling.so
#16 0x0000100002e0d500 in TClingCallFunc::IFacePtr() () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-09-2300/external/el8_ppc64le_gcc12/lib/libCling.so
#17 0x0000100002d6676c in TCling::CallFunc_IFacePtr(CallFunc_t*) const () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/cms/cmssw/CMSSW_14_0_X_2024-01-09-2300/external/el8_ppc64le_gcc12/lib/libCling.so
#18 0x00001000011c04cc in WrapperCall(long, unsigned long, void*, void*, void*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libcppyy_backend3_9.so
#19 0x00001000011c1e58 in Cppyy::CallO(long, void*, unsigned long, void*, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libcppyy_backend3_9.so
#20 0x0000100001129be4 in CPyCppyy::(anonymous namespace)::InstanceExecutor::Execute(long, void*, CPyCppyy::CallContext*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libcppyy3_9.so
#21 0x000010000110a7c8 in CPyCppyy::CPPMethod::ExecuteFast(void*, long, CPyCppyy::CallContext*) [clone .isra.0] () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libcppyy3_9.so
#22 0x000010000110c888 in CPyCppyy::CPPMethod::Execute(void*, long, CPyCppyy::CallContext*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libcppyy3_9.so
#23 0x000010000110da60 in CPyCppyy::CPPMethod::Call(CPyCppyy::CPPInstance*&, _object*, _object*, CPyCppyy::CallContext*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libcppyy3_9.so
#24 0x000010000111431c in CPyCppyy::(anonymous namespace)::mp_call(CPyCppyy::CPPOverload*, _object*, _object*) () from /cvmfs/cms-ib.cern.ch/sw/ppc64le/nweek-02819/el8_ppc64le_gcc12/lcg/root/6.30.03-889fd1c5cd05d2115512394e77c1cb76/lib/libcppyy3_9.so
#25 0x00001000001676ec in _PyObject_Call (tstate=0x100012a9f60, callable=0x10001f818430, args=0x1000009d4040, kwargs=<optimized out>) at Objects/call.c:281
#26 0x00001000001fd714 in slot_tp_call (self=0x100020096bc0, args=0x1000009d4040, kwds=0x0) at Objects/typeobject.c:6734
#27 0x0000100000167d10 in _PyObject_MakeTpCall (tstate=0x100012a9f60, callable=0x100020096bc0, args=0x1000086c31e8, nargs=<optimized out>, keywords=0x0) at Objects/call.c:191
#28 0x00001000000fcc98 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775808, args=0x1000086c31e8, callable=0x100020096bc0, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:116
#29 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775808, args=0x1000086c31e8, callable=0x100020096bc0, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:103
#30 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775808, args=0x1000086c31e8, callable=0x100020096bc0) at ./Include/cpython/abstract.h:127
#31 call_function (kwnames=0x0, oparg=0, pp_stack=<synthetic pointer>, tstate=0x100012a9f60) at Python/ceval.c:5077
#32 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x1000086c3040, throwflag=<optimized out>) at Python/ceval.c:3489
#33 0x00001000002c2cc0 in _PyEval_EvalFrame (throwflag=0, f=0x1000086c3040, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#34 _PyEval_EvalCode (tstate=0x100012a9f60, _co=0x10001f4edf50, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x100000fc35d8, kwcount=0, kwstep=1, defs=0x100000ea2f28, defcount=1, kwdefs=0x0, closure=0x0, name=0x1000009d0530, qualname=0x10001f4e4210) at Python/ceval.c:4329
#35 0x000010000016797c in _PyFunction_Vectorcall (func=<optimized out>, stack=0x100000fc35b8, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#36 0x00001000000fb36c in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775812, args=0x100000fc35b8, callable=0x10002008b160, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:118
#37 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775812, args=0x100000fc35b8, callable=0x10002008b160) at ./Include/cpython/abstract.h:127
#38 call_function (kwnames=0x0, oparg=4, pp_stack=<synthetic pointer>, tstate=0x100012a9f60) at Python/ceval.c:5077
#39 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x100000fc3420, throwflag=<optimized out>) at Python/ceval.c:3489
#40 0x00001000002c2cc0 in _PyEval_EvalFrame (throwflag=0, f=0x100000fc3420, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#41 _PyEval_EvalCode (tstate=0x100012a9f60, _co=0x100000f4b660, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x3fffd2a54798, kwcount=0, kwstep=1, defs=0x100000ea5b08, defcount=1, kwdefs=0x0, closure=0x0, name=0x1000009d0530, qualname=0x100000e37da0) at Python/ceval.c:4329
#42 0x000010000016797c in _PyFunction_Vectorcall (func=<optimized out>, stack=0x3fffd2a54790, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#43 0x0000100000167f64 in _PyObject_FastCallDictTstate (tstate=0x100012a9f60, callable=0x100000e66ca0, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at Objects/call.c:118
#44 0x00001000001683b0 in _PyObject_Call_Prepend (tstate=0x100012a9f60, callable=0x100000e66ca0, obj=0x100000cd5fd0, args=0x1000009d4040, kwargs=0x0) at Objects/call.c:489
#45 0x0000100000201efc in slot_tp_init (self=0x100000cd5fd0, args=0x1000009d4040, kwds=0x0) at Objects/typeobject.c:6971
#46 0x00001000001f5900 in type_call (type=0x10008190f30, args=0x1000009d4040, kwds=0x0) at Objects/typeobject.c:1028
#47 0x0000100000167d10 in _PyObject_MakeTpCall (tstate=0x100012a9f60, callable=0x10008190f30, args=0x100000e34a40, nargs=<optimized out>, keywords=0x0) at Objects/call.c:191
#48 0x00001000000fb9f8 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775808, args=0x100000e34a40, callable=0x10008190f30, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:116
#49 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775808, args=0x100000e34a40, callable=0x10008190f30, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:103
#50 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775808, args=0x100000e34a40, callable=0x10008190f30) at ./Include/cpython/abstract.h:127
#51 call_function (kwnames=0x0, oparg=0, pp_stack=<synthetic pointer>, tstate=0x100012a9f60) at Python/ceval.c:5077
#52 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x100000e348b0, throwflag=<optimized out>) at Python/ceval.c:3520
#53 0x00001000000f315c in _PyEval_EvalFrame (throwflag=0, f=<optimized out>, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#54 function_code_fastcall (tstate=0x100012a9f60, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at Objects/call.c:330
#55 0x000010000016bfb0 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=0x100000dbf8b0, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:118
#56 method_vectorcall (method=<optimized out>, args=0x100000ea6b80, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/classobject.c:53
#57 0x00001000000fa4ac in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775808, args=0x100000ea6b80, callable=0x100000ed6800, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:118
#58 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775808, args=0x100000ea6b80, callable=0x100000ed6800) at ./Include/cpython/abstract.h:127
#59 call_function (kwnames=0x0, oparg=0, pp_stack=<synthetic pointer>, tstate=0x100012a9f60) at Python/ceval.c:5077
#60 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x100000ea6a00, throwflag=<optimized out>) at Python/ceval.c:3520
#61 0x00001000000f315c in _PyEval_EvalFrame (throwflag=0, f=<optimized out>, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#62 function_code_fastcall (tstate=0x100012a9f60, co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:330
#63 0x00001000000fb24c in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775810, args=<optimized out>, callable=0x100000e24280, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:118
#64 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775810, args=<optimized out>, callable=0x100000e24280) at ./Include/cpython/abstract.h:127
#65 call_function (kwnames=0x0, oparg=2, pp_stack=<synthetic pointer>, tstate=0x100012a9f60) at Python/ceval.c:5077
#66 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x10001540500, throwflag=<optimized out>) at Python/ceval.c:3506
#67 0x00001000002c2cc0 in _PyEval_EvalFrame (throwflag=0, f=0x10001540500, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#68 _PyEval_EvalCode (tstate=0x100012a9f60, _co=0x100000d8ba80, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x3fffd2a55130, kwcount=0, kwstep=1, defs=0x100000d359e8, defcount=1, kwdefs=0x0, closure=0x0, name=0x100000a15b30, qualname=0x100000d89270) at Python/ceval.c:4329
#69 0x000010000016797c in _PyFunction_Vectorcall (func=<optimized out>, stack=0x3fffd2a55120, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#70 0x000010000016beb8 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x3fffd2a55120, callable=0x100000e24430, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:118
#71 method_vectorcall (method=<optimized out>, args=0x100000db5fe8, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:83
#72 0x0000100000167490 in PyVectorcall_Call (callable=0x100000ed6840, tuple=<error reading variable: value has been optimized out>, kwargs=<optimized out>) at Objects/call.c:231
#73 0x000010000016778c in _PyObject_Call (tstate=0x100012a9f60, callable=0x100000ed6840, args=0x100000db5fd0, kwargs=<optimized out>) at Objects/call.c:266
#74 0x0000100000167824 in PyObject_Call () at Objects/call.c:290
#75 0x00001000000f8ee4 in do_call_core (kwdict=0x100000ed6900, callargs=0x100000db5fd0, func=0x100000ed6840, tstate=<optimized out>) at Python/ceval.c:5125
#76 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x100000dc7e40, throwflag=<optimized out>) at Python/ceval.c:3582
#77 0x00001000002c2cc0 in _PyEval_EvalFrame (throwflag=0, f=0x100000dc7e40, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#78 _PyEval_EvalCode (tstate=0x100012a9f60, _co=0x100000d8bc90, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x3fffd2a556f0, kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x1000009d01f0, qualname=0x100000d8a670) at Python/ceval.c:4329
#79 0x000010000016797c in _PyFunction_Vectorcall (func=<optimized out>, stack=0x3fffd2a556e0, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#80 0x0000100000167f64 in _PyObject_FastCallDictTstate (tstate=0x100012a9f60, callable=0x100000e245e0, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at Objects/call.c:118
#81 0x00001000001683b0 in _PyObject_Call_Prepend (tstate=0x100012a9f60, callable=0x100000e245e0, obj=0x100000cd5f70, args=0x100000ea2ee0, kwargs=0x0) at Objects/call.c:489
#82 0x00001000001fd784 in slot_tp_call (self=0x100000cd5f70, args=0x100000ea2ee0, kwds=0x0) at Objects/typeobject.c:6731
#83 0x0000100000167d10 in _PyObject_MakeTpCall (tstate=0x100012a9f60, callable=0x100000cd5f70, args=0x100000e5bf08, nargs=<optimized out>, keywords=0x0) at Objects/call.c:191
#84 0x00001000000fb9f8 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x100000e5bf08, callable=0x100000cd5f70, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:116
#85 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x100000e5bf08, callable=0x100000cd5f70, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:103
#86 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x100000e5bf08, callable=0x100000cd5f70) at ./Include/cpython/abstract.h:127
#87 call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>, tstate=0x100012a9f60) at Python/ceval.c:5077
#88 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x100000e5bd60, throwflag=<optimized out>) at Python/ceval.c:3520
#89 0x00001000002c2cc0 in _PyEval_EvalFrame (throwflag=0, f=0x100000e5bd60, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#90 _PyEval_EvalCode (tstate=0x100012a9f60, _co=0x100000e19500, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x3fffd2a55bb0, kwcount=0, kwstep=1, defs=0x100000b46d48, defcount=1, kwdefs=0x0, closure=0x0, name=0x100000a15b30, qualname=0x100000d958b0) at Python/ceval.c:4329
#91 0x000010000016797c in _PyFunction_Vectorcall (func=<optimized out>, stack=0x3fffd2a55ba0, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#92 0x000010000016beb8 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x3fffd2a55ba0, callable=0x100000e0a430, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:118
#93 method_vectorcall (method=<optimized out>, args=0x100000e463e8, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:83
#94 0x0000100000167490 in PyVectorcall_Call (callable=0x100000ed6b00, tuple=<error reading variable: value has been optimized out>, kwargs=<optimized out>) at Objects/call.c:231
#95 0x000010000016778c in _PyObject_Call (tstate=0x100012a9f60, callable=0x100000ed6b00, args=0x100000e463d0, kwargs=<optimized out>) at Objects/call.c:266
#96 0x0000100000167824 in PyObject_Call () at Objects/call.c:290
#97 0x00001000000f8ee4 in do_call_core (kwdict=0x100000e1da00, callargs=0x100000e463d0, func=0x100000ed6b00, tstate=<optimized out>) at Python/ceval.c:5125
#98 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x100000dc7740, throwflag=<optimized out>) at Python/ceval.c:3582
#99 0x00001000002c2cc0 in _PyEval_EvalFrame (throwflag=0, f=0x100000dc7740, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#100 _PyEval_EvalCode (tstate=0x100012a9f60, _co=0x100000d773a0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x3fffd2a56170, kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x1000009d01f0, qualname=0x100000e16080) at Python/ceval.c:4329
#101 0x000010000016797c in _PyFunction_Vectorcall (func=<optimized out>, stack=0x3fffd2a56160, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#102 0x0000100000167f64 in _PyObject_FastCallDictTstate (tstate=0x100012a9f60, callable=0x100000e0a310, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at Objects/call.c:118
#103 0x00001000001683b0 in _PyObject_Call_Prepend (tstate=0x100012a9f60, callable=0x100000e0a310, obj=0x100000cd5f10, args=0x100000abe9d0, kwargs=0x0) at Objects/call.c:489
#104 0x00001000001fd784 in slot_tp_call (self=0x100000cd5f10, args=0x100000abe9d0, kwds=0x0) at Objects/typeobject.c:6731
#105 0x0000100000167d10 in _PyObject_MakeTpCall (tstate=0x100012a9f60, callable=0x100000cd5f10, args=0x100000e5bd28, nargs=<optimized out>, keywords=0x0) at Objects/call.c:191
#106 0x00001000000fb9f8 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x100000e5bd28, callable=0x100000cd5f10, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:116
#107 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x100000e5bd28, callable=0x100000cd5f10, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:103
#108 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x100000e5bd28, callable=0x100000cd5f10) at ./Include/cpython/abstract.h:127
#109 call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>, tstate=0x100012a9f60) at Python/ceval.c:5077
#110 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x100000e5bb80, throwflag=<optimized out>) at Python/ceval.c:3520
#111 0x00001000002c2cc0 in _PyEval_EvalFrame (throwflag=0, f=0x100000e5bb80, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#112 _PyEval_EvalCode (tstate=0x100012a9f60, _co=0x100000e19500, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x3fffd2a56630, kwcount=0, kwstep=1, defs=0x100000b46d48, defcount=1, kwdefs=0x0, closure=0x0, name=0x100000a15b30, qualname=0x100000d958b0) at Python/ceval.c:4329
#113 0x000010000016797c in _PyFunction_Vectorcall (func=<optimized out>, stack=0x3fffd2a56620, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#114 0x000010000016beb8 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=<optimized out>, args=0x3fffd2a56620, callable=0x100000e0a430, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:118
#115 method_vectorcall (method=<optimized out>, args=0x100000cdbbc8, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:83
#116 0x0000100000167490 in PyVectorcall_Call (callable=0x100000ed6940, tuple=<error reading variable: value has been optimized out>, kwargs=<optimized out>) at Objects/call.c:231
#117 0x000010000016778c in _PyObject_Call (tstate=0x100012a9f60, callable=0x100000ed6940, args=0x100000cdbbb0, kwargs=<optimized out>) at Objects/call.c:266
#118 0x0000100000167824 in PyObject_Call () at Objects/call.c:290
#119 0x00001000000f8ee4 in do_call_core (kwdict=0x100000ed6ac0, callargs=0x100000cdbbb0, func=0x100000ed6940, tstate=<optimized out>) at Python/ceval.c:5125
#120 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x100000dc7580, throwflag=<optimized out>) at Python/ceval.c:3582
#121 0x00001000002c2cc0 in _PyEval_EvalFrame (throwflag=0, f=0x100000dc7580, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#122 _PyEval_EvalCode (tstate=0x100012a9f60, _co=0x100000d773a0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x3fffd2a56bf0, kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x1000009d01f0, qualname=0x100000e16080) at Python/ceval.c:4329
#123 0x000010000016797c in _PyFunction_Vectorcall (func=<optimized out>, stack=0x3fffd2a56be0, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#124 0x0000100000167f64 in _PyObject_FastCallDictTstate (tstate=0x100012a9f60, callable=0x100000e0a310, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at Objects/call.c:118
#125 0x00001000001683b0 in _PyObject_Call_Prepend (tstate=0x100012a9f60, callable=0x100000e0a310, obj=0x100000cd5640, args=0x100000b782e0, kwargs=0x0) at Objects/call.c:489
#126 0x00001000001fd784 in slot_tp_call (self=0x100000cd5640, args=0x100000b782e0, kwds=0x0) at Objects/typeobject.c:6731
#127 0x0000100000167d10 in _PyObject_MakeTpCall (tstate=0x100012a9f60, callable=0x100000cd5640, args=0x100013bf3c8, nargs=<optimized out>, keywords=0x0) at Objects/call.c:191
#128 0x00001000000fb9f8 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x100013bf3c8, callable=0x100000cd5640, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:116
#129 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=0x100013bf3c8, callable=0x100000cd5640, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:103
#130 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=0x100013bf3c8, callable=0x100000cd5640) at ./Include/cpython/abstract.h:127
#131 call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>, tstate=0x100012a9f60) at Python/ceval.c:5077
#132 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x100013bf1d0, throwflag=<optimized out>) at Python/ceval.c:3520
#133 0x00001000000f315c in _PyEval_EvalFrame (throwflag=0, f=<optimized out>, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#134 function_code_fastcall (tstate=0x100012a9f60, co=<optimized out>, args=<optimized out>, nargs=2, globals=<optimized out>) at Objects/call.c:330
#135 0x00001000000fb24c in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775810, args=<optimized out>, callable=0x100000dbf0d0, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:118
#136 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775810, args=<optimized out>, callable=0x100000dbf0d0) at ./Include/cpython/abstract.h:127
#137 call_function (kwnames=0x0, oparg=2, pp_stack=<synthetic pointer>, tstate=0x100012a9f60) at Python/ceval.c:5077
#138 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x100000e5b5e0, throwflag=<optimized out>) at Python/ceval.c:3506
#139 0x00001000000f315c in _PyEval_EvalFrame (throwflag=0, f=<optimized out>, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#140 function_code_fastcall (tstate=0x100012a9f60, co=<optimized out>, args=<optimized out>, nargs=1, globals=<optimized out>) at Objects/call.c:330
#141 0x00001000000fb24c in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=9223372036854775809, args=<optimized out>, callable=0x100000dbf790, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:118
#142 PyObject_Vectorcall (kwnames=0x0, nargsf=9223372036854775809, args=<optimized out>, callable=0x100000dbf790) at ./Include/cpython/abstract.h:127
#143 call_function (kwnames=0x0, oparg=1, pp_stack=<synthetic pointer>, tstate=0x100012a9f60) at Python/ceval.c:5077
#144 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x10001429320, throwflag=<optimized out>) at Python/ceval.c:3506
#145 0x00001000002c2cc0 in _PyEval_EvalFrame (throwflag=0, f=0x10001429320, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#146 _PyEval_EvalCode (tstate=0x100012a9f60, _co=0x100000e0fea0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kwnames=0x100000b78ad8, kwargs=0x100000d4c120, kwcount=1, kwstep=1, defs=0x100000e18ad8, defcount=11, kwdefs=0x100000e1da40, closure=0x0, name=0x1000009d0530, qualname=0x100000e01260) at Python/ceval.c:4329
#147 0x000010000016797c in _PyFunction_Vectorcall (func=<optimized out>, stack=0x100000d4c118, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:396
#148 0x0000100000168038 in _PyObject_FastCallDictTstate (tstate=0x100012a9f60, callable=0x100000dbf1f0, args=<optimized out>, nargsf=<optimized out>, kwargs=<optimized out>) at Objects/call.c:129
#149 0x00001000001683b0 in _PyObject_Call_Prepend (tstate=0x100012a9f60, callable=0x100000dbf1f0, obj=0x100000adbdf0, args=0x1000009d4040, kwargs=0x100000af3240) at Objects/call.c:489
#150 0x0000100000201efc in slot_tp_init (self=0x100000adbdf0, args=0x1000009d4040, kwds=0x100000af3240) at Objects/typeobject.c:6971
#151 0x00001000001f5900 in type_call (type=0x100013bd8d0, args=0x1000009d4040, kwds=0x100000af3240) at Objects/typeobject.c:1028
#152 0x0000100000167d10 in _PyObject_MakeTpCall (tstate=0x100012a9f60, callable=0x100013bd8d0, args=0x10001366db0, nargs=<optimized out>, keywords=0x100000b78a30) at Objects/call.c:191
#153 0x00001000000fb3a4 in _PyObject_VectorcallTstate (kwnames=0x100000b78a30, nargsf=9223372036854775808, args=<optimized out>, callable=0x100013bd8d0, tstate=<optimized out>) at ./Include/cpython/abstract.h:116
#154 _PyObject_VectorcallTstate (kwnames=0x100000b78a30, nargsf=9223372036854775808, args=<optimized out>, callable=0x100013bd8d0, tstate=0x100012a9f60) at ./Include/cpython/abstract.h:103
#155 PyObject_Vectorcall (kwnames=0x100000b78a30, nargsf=9223372036854775808, args=<optimized out>, callable=0x100013bd8d0) at ./Include/cpython/abstract.h:127
#156 call_function (kwnames=0x100000b78a30, oparg=<optimized out>, pp_stack=<synthetic pointer>, tstate=<optimized out>) at Python/ceval.c:5077
#157 _PyEval_EvalFrameDefault (tstate=0x100012a9f60, f=0x10001366c40, throwflag=<optimized out>) at Python/ceval.c:3537
#158 0x00001000002c2cc0 in _PyEval_EvalFrame (throwflag=0, f=0x10001366c40, tstate=0x100012a9f60) at ./Include/internal/pycore_ceval.h:40
#159 _PyEval_EvalCode (tstate=0x100012a9f60, _co=_co
entry=0x100000b20660, globals=globals
entry=0x100000af7680, locals=locals
entry=0x100000af7680, args=args
entry=0x0, argcount=<optimized out>, kwnames=0x0, kwargs=0x0, kwcount=0, kwstep=kwstep
entry=2, defs=defs
entry=0x0, defcount=defcount
entry=0, kwdefs=kwdefs
entry=0x0, closure=closure
entry=0x0, name=name
entry=0x0, qualname=qualname
entry=0x0) at Python/ceval.c:4329
#160 0x00001000002c31a4 in _PyEval_EvalCodeWithName (_co=0x100000b20660, globals=0x100000af7680, locals=0x100000af7680, args=0x0, argcount=<optimized out>, kwnames=<optimized out>, kwargs=<optimized out>, kwcount=<optimized out>, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:4361
#161 0x00001000002c3234 in PyEval_EvalCodeEx (_co=<optimized out>, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=<optimized out>, kws=<optimized out>, kwcount=<optimized out>, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0) at Python/ceval.c:4377
#162 0x00001000002c32ac in PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at Python/ceval.c:828
#163 0x000010000031b584 in run_eval_code_obj (locals=0x100000af7680, globals=0x100000af7680, co=0x100000b20660, tstate=0x100012a9f60) at Python/pythonrun.c:1221
#164 run_mod (mod=mod
entry=0x10001478b70, filename=filename
entry=0x100000b856f0, globals=globals
entry=0x100000af7680, locals=locals
entry=0x100000af7680, flags=flags
entry=0x3fffd2a57dd0, arena=arena
entry=0x100000a12950) at Python/pythonrun.c:1242
#165 0x000010000031dce4 in pyrun_file (flags=0x3fffd2a57dd0, closeit=<optimized out>, locals=0x100000af7680, globals=0x100000af7680, start=257, filename=0x100000b856f0, fp=0x100012a3430) at Python/pythonrun.c:1140
#166 pyrun_simple_file (flags=0x3fffd2a57dd0, closeit=<optimized out>, filename=0x100000b856f0, fp=0x100012a3430) at Python/pythonrun.c:450
#167 PyRun_SimpleFileExFlags (fp=0x100012a3430, filename=<optimized out>, closeit=<optimized out>, flags=0x3fffd2a57dd0) at Python/pythonrun.c:483
#168 0x000010000031e474 in PyRun_AnyFileExFlags (fp=0x100012a3430, filename=0x100000b869f0 "/scratch/cmsbuild/jenkins_b/workspace/ib-run-qa/CMSSW_14_0_X_2024-01-09-2300/src/PhysicsTools/SelectorUtils/test/test_vid_selectors.py", closeit=<optimized out>, flags=0x3fffd2a57dd0) at Python/pythonrun.c:92
#169 0x00001000003474ec in pymain_run_file (cf=<optimized out>, config=0x100012abb10) at Modules/main.c:373
#170 pymain_run_python (exitcode=0x3fffd2a57ddc) at Modules/main.c:598
#171 Py_RunMain () at Modules/main.c:677
#172 0x0000100000347c5c in pymain_main (args=0x3fffd2a57e80) at Modules/main.c:707
#173 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:731
#174 0x0000000010000718 in main (argc=<optimized out>, argv=<optimized out>) at ./Programs/python.c:15
===========================================================

Crash seems to happen during execution of this line: https://github.com/cms-sw/cmssw/blob/master/PhysicsTools/SelectorUtils/test/test_vid_selectors.py#L12

These two lines at the end of the log could also be of interest:

pure virtual method called
terminate called without an active exception
cmsbuild commented 8 months ago

cms-bot internal usage

cmsbuild commented 8 months ago

A new Issue was created by @iarspider .

@smuzaffar, @Dr15Jones, @makortel, @sextonkennedy, @antoniovilela, @rappoccio can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

iarspider commented 8 months ago

assign PhysicsTools/SelectorUtils

cmsbuild commented 8 months ago

New categories assigned: reconstruction

@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

makortel commented 8 months ago

The crash comes via cling @vgvassilev @pcanal

makortel commented 8 months ago

assign core

makortel commented 8 months ago

type root

cmsbuild commented 8 months ago

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

vgvassilev commented 8 months ago

That looks like a stack overwrite. Would root-project/root#13850 resolve this problem?

iarspider commented 7 months ago

Happened again in (CMSSW_14_1_X_2024-02-07-2300)[https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_ppc64le_gcc12/CMSSW_14_1_X_2024-02-07-2300/unitTestLogs/PhysicsTools/SelectorUtils#/811-811], with ROOT v6.30.05 + a custom patch to "Have TBasket::ReadBasketBuffers return different values for each of the different error cases".

vgvassilev commented 7 months ago

I think we should try getting https://github.com/root-project/root/pull/13850 in rather than hunting down these issues as that part of llvm is not well maintained...

makortel commented 3 months ago

Given that we dropped PowerPC support, could we close this issue? @smuzaffar what do you think?

makortel commented 2 weeks ago

-core

PowerPC support has been dropped

makortel commented 2 weeks ago

@cmsbuild, please close