Closed rubihason closed 1 year ago
Can you check if it also crashes with a single thread? Can you recompile with debug symbols to get a more informative backtrack, ideally from a debugger?
This is while i run for a single thread:
10:43:19 RootVertexPe INFO Total number of reco track-associated truth primary vertices : 1
10:43:19 TracksToTraj ERROR Last trajectory is empty
10:43:19 CsvMultiTraj WARNING Empty multiTrajectory.
*** Break *** segmentation violation
===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0 0x00007f94de911c3a in __GI___wait4 (pid=80, stat_loc=stat_loc
entry=0x7ffc6b2f4768, options=options
entry=0, usage=usage
entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
#1 0x00007f94de911bfb in __GI___waitpid (pid=<optimized out>, stat_loc=stat_loc
entry=0x7ffc6b2f4768, options=options
entry=0) at waitpid.c:38
#2 0x00007f94de880f67 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:172
#3 0x00007f94dcf8b4be in TUnixSystem::StackTrace() () from /root/Thesis/Root/root_install/lib/libCore.so
#4 0x00007f94dcf88345 in TUnixSystem::DispatchSignals(ESignals) () from /root/Thesis/Root/root_install/lib/libCore.so
#5 <signal handler called>
#6 0x00007f94dd8ff45c in ActsExamples::AmbiguityResolutionAlgorithm::execute(ActsExamples::AlgorithmContext const&) const () from /root/Thesis/acts/acts.build/lib/libActsExamplesTrackFinding.so
#7 0x00007f94dd216da1 in ActsExamples::Sequencer::run()::{lambda()#1}::operator()() const::{lambda(tbb::blocked_range<unsigned long> const&)#1}::operator()(tbb::blocked_range<unsigned long> const&) const () from /root/Thesis/acts/acts.build/lib/libActsExamplesFramework.so
#8 0x00007f94dd21e6be in ActsExamples::Sequencer::run() () from /root/Thesis/acts/acts.build/lib/libActsExamplesFramework.so
#9 0x00007f94ddf2f1a3 in ?? () from /root/Thesis/acts/acts.build/python/acts/ActsPythonBindings.cpython-38-x86_64-linux-gnu.so
#10 0x00007f94ddf475ee in ?? () from /root/Thesis/acts/acts.build/python/acts/ActsPythonBindings.cpython-38-x86_64-linux-gnu.so
#11 0x00000000005f6929 in PyCFunction_Call ()
#12 0x00000000005f74f6 in _PyObject_MakeTpCall ()
#13 0x000000000050c383 in ?? ()
#14 0x0000000000570b26 in _PyEval_EvalFrameDefault ()
#15 0x0000000000569dba in _PyEval_EvalCodeWithName ()
#16 0x00000000006902a7 in PyEval_EvalCode ()
#17 0x000000000067f951 in ?? ()
#18 0x000000000067f9cf in ?? ()
#19 0x000000000067fa71 in ?? ()
#20 0x0000000000681b97 in PyRun_SimpleFileExFlags ()
#21 0x00000000006b9d32 in Py_RunMain ()
#22 0x00000000006ba0bd in Py_BytesMain ()
#23 0x00007f94de853083 in __libc_start_main (main=0x4efd60 <main>, argc=2, argv=0x7ffc6b2f8f08, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc6b2f8ef8) at ../csu/libc-start.c:308
#24 0x00000000005fc5fe in _start ()
===========================================================
The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum https://root.cern/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#6 0x00007f94dd8ff45c in ActsExamples::AmbiguityResolutionAlgorithm::execute(ActsExamples::AlgorithmContext const&) const () from /root/Thesis/acts/acts.build/lib/libActsExamplesTrackFinding.so
#7 0x00007f94dd216da1 in ActsExamples::Sequencer::run()::{lambda()#1}::operator()() const::{lambda(tbb::blocked_range<unsigned long> const&)#1}::operator()(tbb::blocked_range<unsigned long> const&) const () from /root/Thesis/acts/acts.build/lib/libActsExamplesFramework.so
#8 0x00007f94dd21e6be in ActsExamples::Sequencer::run() () from /root/Thesis/acts/acts.build/lib/libActsExamplesFramework.so
#9 0x00007f94ddf2f1a3 in ?? () from /root/Thesis/acts/acts.build/python/acts/ActsPythonBindings.cpython-38-x86_64-linux-gnu.so
#10 0x00007f94ddf475ee in ?? () from /root/Thesis/acts/acts.build/python/acts/ActsPythonBindings.cpython-38-x86_64-linux-gnu.so
#11 0x00000000005f6929 in PyCFunction_Call ()
#12 0x00000000005f74f6 in _PyObject_MakeTpCall ()
#13 0x000000000050c383 in ?? ()
#14 0x0000000000570b26 in _PyEval_EvalFrameDefault ()
#15 0x0000000000569dba in _PyEval_EvalCodeWithName ()
#16 0x00000000006902a7 in PyEval_EvalCode ()
#17 0x000000000067f951 in ?? ()
#18 0x000000000067f9cf in ?? ()
#19 0x000000000067fa71 in ?? ()
#20 0x0000000000681b97 in PyRun_SimpleFileExFlags ()
#21 0x00000000006b9d32 in Py_RunMain ()
#22 0x00000000006ba0bd in Py_BytesMain ()
#23 0x00007f94de853083 in __libc_start_main (main=0x4efd60 <main>, argc=2, argv=0x7ffc6b2f8f08, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc6b2f8ef8) at ../csu/libc-start.c:308
#24 0x00000000005fc5fe in _start ()
===========================================================
And this is one thread with logging.VERBOSE
14:47:37 TracksToTraj ERROR Last trajectory is empty
14:47:37 EventStore#1 VERBOSE Added object 'trajectories-from-selected-tracks' of type St6vectorIN12ActsExamples12TrajectoriesESaIS1_EE
14:47:37 EventStore#1 VERBOSE Added alias object 'trajectories'
14:47:37 Sequencer VERBOSE Execute Writer: RootTrajectoryStatesWriter
14:47:37 EventStore#1 VERBOSE Attempt to get object 'trajectories-from-tracks' of type St6vectorIN12ActsExamples12TrajectoriesESaIS1_EE
14:47:37 EventStore#1 VERBOSE Retrieved object 'trajectories-from-tracks'
14:47:37 EventStore#1 VERBOSE Attempt to get object 'particles_selected' of type N5boost9container8flat_setIN10ActsFatras8ParticleEN12ActsExamples6detail17CompareParticleIdEvEE
14:47:37 EventStore#1 VERBOSE Retrieved object 'particles_selected'
14:47:37 EventStore#1 VERBOSE Attempt to get object 'simhits' of type N5boost9container13flat_multisetIN10ActsFatras3HitEN12ActsExamples6detail17CompareGeometryIdEvEE
14:47:37 EventStore#1 VERBOSE Retrieved object 'simhits'
14:47:37 EventStore#1 VERBOSE Attempt to get object 'measurement_particles_map' of type N5boost9container13flat_multimapIjN10ActsFatras7BarcodeESt4lessIjEvEE
14:47:37 EventStore#1 VERBOSE Retrieved object 'measurement_particles_map'
14:47:37 EventStore#1 VERBOSE Attempt to get object 'measurement_simhits_map' of type N5boost9container13flat_multimapIjjSt4lessIjEvEE
14:47:37 EventStore#1 VERBOSE Retrieved object 'measurement_simhits_map'
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|27|0|0=4503600080355328
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|27|0|0=4503600080355328
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 Sequencer VERBOSE Execute Writer: RootTrajectorySummaryWriter
14:47:37 EventStore#1 VERBOSE Attempt to get object 'trajectories-from-tracks' of type St6vectorIN12ActsExamples12TrajectoriesESaIS1_EE
14:47:37 EventStore#1 VERBOSE Retrieved object 'trajectories-from-tracks'
14:47:37 EventStore#1 VERBOSE Attempt to get object 'particles_selected' of type N5boost9container8flat_setIN10ActsFatras8ParticleEN12ActsExamples6detail17CompareParticleIdEvEE
14:47:37 EventStore#1 VERBOSE Retrieved object 'particles_selected'
14:47:37 EventStore#1 VERBOSE Attempt to get object 'measurement_particles_map' of type N5boost9container13flat_multimapIjN10ActsFatras7BarcodeESt4lessIjEvEE
14:47:37 EventStore#1 VERBOSE Retrieved object 'measurement_particles_map'
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|27|0|0=4503600080355328
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|27|0|0=4503600080355328
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|22|0|0=4503599996469248
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 RootTrajecto VERBOSE Find the truth particle with barcode 1|0|24|0|0=4503600030023680
14:47:37 Sequencer VERBOSE Execute Writer: CKFPerformanceWriter
14:47:37 EventStore#1 VERBOSE Attempt to get object 'trajectories-from-tracks' of type St6vectorIN12ActsExamples12TrajectoriesESaIS1_EE
14:47:37 EventStore#1 VERBOSE Retrieved object 'trajectories-from-tracks'
14:47:37 EventStore#1 VERBOSE Attempt to get object 'truth_seeds_selected' of type N5boost9container8flat_setIN10ActsFatras8ParticleEN12ActsExamples6detail17CompareParticleIdEvEE
14:47:37 EventStore#1 VERBOSE Retrieved object 'truth_seeds_selected'
14:47:37 EventStore#1 VERBOSE Attempt to get object 'measurement_particles_map' of type N5boost9container13flat_multimapIjN10ActsFatras7BarcodeESt4lessIjEvEE
14:47:37 EventStore#1 VERBOSE Retrieved object 'measurement_particles_map'
14:47:37 Sequencer VERBOSE Execute Writer: CsvMultiTrajectoryWriter
14:47:37 EventStore#1 VERBOSE Attempt to get object 'trajectories-from-tracks' of type St6vectorIN12ActsExamples12TrajectoriesESaIS1_EE
14:47:37 EventStore#1 VERBOSE Retrieved object 'trajectories-from-tracks'
14:47:37 EventStore#1 VERBOSE Attempt to get object 'measurement_particles_map' of type N5boost9container13flat_multimapIjN10ActsFatras7BarcodeESt4lessIjEvEE
14:47:37 EventStore#1 VERBOSE Retrieved object 'measurement_particles_map'
14:47:37 Sequencer VERBOSE Execute Algorithm: AmbiguityResolutionAlgorithm
14:47:37 EventStore#1 VERBOSE Attempt to get object 'selectedTracks' of type N4Acts14TrackContainerINS_25ConstVectorTrackContainerENS_26ConstVectorMultiTrajectoryESt10shared_ptrEE
14:47:37 EventStore#1 VERBOSE Retrieved object 'selectedTracks'
*** Break *** segmentation violation
===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0 0x00007fb9661afc3a in __GI___wait4 (pid=7203, stat_loc=stat_loc
entry=0x7ffe5e725de8, options=options
entry=0, usage=usage
entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
#1 0x00007fb9661afbfb in __GI___waitpid (pid=<optimized out>, stat_loc=stat_loc
entry=0x7ffe5e725de8, options=options
entry=0) at waitpid.c:38
#2 0x00007fb96611ef67 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:172
#3 0x00007fb9648004be in TUnixSystem::StackTrace() () from /root/Thesis/Root/root_install/lib/libCore.so
#4 0x00007fb9647fd345 in TUnixSystem::DispatchSignals(ESignals) () from /root/Thesis/Root/root_install/lib/libCore.so
#5 <signal handler called>
#6 0x00007fb965187fec in ActsExamples::AmbiguityResolutionAlgorithm::execute(ActsExamples::AlgorithmContext const&) const () from /root/Thesis/acts/acts.build/lib/libActsExamplesTrackFinding.so
#7 0x00007fb964a8bda1 in ActsExamples::Sequencer::run()::{lambda()#1}::operator()() const::{lambda(tbb::blocked_range<unsigned long> const&)#1}::operator()(tbb::blocked_range<unsigned long> const&) const () from /root/Thesis/acts/acts.build/lib/libActsExamplesFramework.so
#8 0x00007fb964a936be in ActsExamples::Sequencer::run() () from /root/Thesis/acts/acts.build/lib/libActsExamplesFramework.so
#9 0x00007fb9657cae13 in ?? () from /root/Thesis/acts/acts.build/python/acts/ActsPythonBindings.cpython-38-x86_64-linux-gnu.so
#10 0x00007fb9657e305e in ?? () from /root/Thesis/acts/acts.build/python/acts/ActsPythonBindings.cpython-38-x86_64-linux-gnu.so
#11 0x00000000005f6929 in PyCFunction_Call ()
#12 0x00000000005f74f6 in _PyObject_MakeTpCall ()
#13 0x000000000050c383 in ?? ()
#14 0x0000000000570b26 in _PyEval_EvalFrameDefault ()
#15 0x0000000000569dba in _PyEval_EvalCodeWithName ()
#16 0x00000000006902a7 in PyEval_EvalCode ()
#17 0x000000000067f951 in ?? ()
#18 0x000000000067f9cf in ?? ()
#19 0x000000000067fa71 in ?? ()
#20 0x0000000000681b97 in PyRun_SimpleFileExFlags ()
#21 0x00000000006b9d32 in Py_RunMain ()
#22 0x00000000006ba0bd in Py_BytesMain ()
#23 0x00007fb9660f1083 in __libc_start_main (main=0x4efd60 <main>, argc=2, argv=0x7ffe5e72a4d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe5e72a4c8) at ../csu/libc-start.c:308
#24 0x00000000005fc5fe in _start ()
===========================================================
The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum https://root.cern/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#6 0x00007fb965187fec in ActsExamples::AmbiguityResolutionAlgorithm::execute(ActsExamples::AlgorithmContext const&) const () from /root/Thesis/acts/acts.build/lib/libActsExamplesTrackFinding.so
#7 0x00007fb964a8bda1 in ActsExamples::Sequencer::run()::{lambda()#1}::operator()() const::{lambda(tbb::blocked_range<unsigned long> const&)#1}::operator()(tbb::blocked_range<unsigned long> const&) const () from /root/Thesis/acts/acts.build/lib/libActsExamplesFramework.so
#8 0x00007fb964a936be in ActsExamples::Sequencer::run() () from /root/Thesis/acts/acts.build/lib/libActsExamplesFramework.so
#9 0x00007fb9657cae13 in ?? () from /root/Thesis/acts/acts.build/python/acts/ActsPythonBindings.cpython-38-x86_64-linux-gnu.so
#10 0x00007fb9657e305e in ?? () from /root/Thesis/acts/acts.build/python/acts/ActsPythonBindings.cpython-38-x86_64-linux-gnu.so
#11 0x00000000005f6929 in PyCFunction_Call ()
#12 0x00000000005f74f6 in _PyObject_MakeTpCall ()
#13 0x000000000050c383 in ?? ()
#14 0x0000000000570b26 in _PyEval_EvalFrameDefault ()
#15 0x0000000000569dba in _PyEval_EvalCodeWithName ()
#16 0x00000000006902a7 in PyEval_EvalCode ()
#17 0x000000000067f951 in ?? ()
#18 0x000000000067f9cf in ?? ()
#19 0x000000000067fa71 in ?? ()
#20 0x0000000000681b97 in PyRun_SimpleFileExFlags ()
#21 0x00000000006b9d32 in Py_RunMain ()
#22 0x00000000006ba0bd in Py_BytesMain ()
#23 0x00007fb9660f1083 in __libc_start_main (main=0x4efd60 <main>, argc=2, argv=0x7ffe5e72a4d8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe5e72a4c8) at ../csu/libc-start.c:308
#24 0x00000000005fc5fe in _start ()
===========================================================
looks like the stack trace is pointing to the ambi solver. could you try to remove it from the chain to confirm @rubihason ?
if that's the case I can take a closer look on Wednesday
I tried to remove the ambi solver from the chain and it worked fine.
can you provide this file @rubihason ?
geometrySelection=srcdir/ "Examples/Algorithms/TrackFinding/share/geoSelection-genericDetector.json"
Ya sure:
[
{
"layer": 2,
"volume": 8
},
{
"layer": 4,
"volume": 8
},
{
"layer": 6,
"volume": 8
},
{
"layer": 8,
"volume": 8
},
{
"layer": 2,
"volume": 9
},
{
"layer": 4,
"volume": 9
},
{
"layer": 6,
"volume": 9
},
{
"layer": 8,
"volume": 9
},
{
"layer": 14,
"volume": 7
},
{
"layer": 12,
"volume": 7
},
{
"layer": 10,
"volume": 7
},
{
"layer": 8,
"volume": 7
}
]
thanks @rubihason
I was not able to run your code directly with main and had to do some changes. which version of acts are you using?
also do you know approximately in which event number the crash happens?
I used the last one, V25.0.0. It is not a constant sometimes he failed in 6k and sometimes around 1k.
I think I was able to reproduce and fix it. can you try https://github.com/acts-project/acts/pull/2070 ? @rubihason
I just ran 10k events, and its works. Thank you very much!
Hello, i running a full chain of Z boson event with CKF and AMVF as follow:
Its works fine when I run it with one hard Scatter and 20 pileups one woke ago, now when i try to running it with only a hard scatter, i get some odd error: