Assert in WaitingTaskList failing during stream end Run

[Dr15Jones]

We have seen several instances of IB RelVal jobs failing with

cmsRun:  [cut]FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

at the stream end Run transition.

[cmsbuild]

A new Issue was created by @Dr15Jones Chris Jones.

@davidlange6, @Dr15Jones, @smuzaffar, @fabiocos can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[Dr15Jones]

A failure: https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc630/CMSSW_10_1_X_2018-02-04-0000/pyRelValMatrixLogs/run/7.4_Cosmics_UP17+Cosmics_UP17+DIGICOSPEAK_UP17+RECOCOSPEAK_UP17+ALCACOS_UP17+HARVESTCOS_UP17/step1_Cosmics_UP17+Cosmics_UP17+DIGICOSPEAK_UP17+RECOCOSPEAK_UP17+ALCACOS_UP17+HARVESTCOS_UP17.log

cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/9c8bdd76c505c251a6b80617921e2b71/opt/cmssw/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Thread 6 (Thread 0x7f5026d3f700 (LWP 172998)):
#0  0x00000036530acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000036530acac0 in sleep () from /lib64/libc.so.6
#2  0x00007f504517b6ee in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00000036530e5419 in syscall () from /lib64/libc.so.6
#5  0x00007f5047f4fc02 in tbb::internal::futex_wait (comparand=2, futex=0x7f5046d790ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f5046d790ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7f5046d790a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f5046d79080) at ../../src/tbb/private_server.cpp:277
#9  0x00007f5047f4fc29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x0000003653407aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x00000036530e8bcd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f50277fe700 (LWP 172997)):
#0  0x00000036530df383 in poll () from /lib64/libc.so.6
#1  0x00007f504517bc84 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f504517c5da in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f504517d806 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000003653032495 in raise () from /lib64/libc.so.6
#6  0x0000003653033c75 in abort () from /lib64/libc.so.6
#7  0x000000365302b60e in __assert_fail_base () from /lib64/libc.so.6
#8  0x000000365302b6d0 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f5049018bce in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/libFWCoreConcurrency.so
#10 0x00007f504938f759 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f5049356fe1 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f5047f5c42c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f5046eb7e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007f5047f54c14 in tbb::internal::arena::process (this=0x7f5046f56d00, s=...) at ../../src/tbb/arena.cpp:160
#14 0x00007f5047f5374b in tbb::internal::market::process (this=0x7f5046f57580, j=...) at ../../src/tbb/market.cpp:693
#15 0x00007f5047f4f9f6 in tbb::internal::rml::private_worker::run (this=0x7f5046d79180) at ../../src/tbb/private_server.cpp:270
#16 0x00007f5047f4fc29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#17 0x0000003653407aa1 in start_thread () from /lib64/libpthread.so.0
#18 0x00000036530e8bcd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f50281ff700 (LWP 172996)):
#0  0x00000036530acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000036530acac0 in sleep () from /lib64/libc.so.6
#2  0x00007f504517b6ee in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00000036530cf4a7 in sched_yield () from /lib64/libc.so.6
#5  0x00007f5047f53763 in tbb::internal::market::process (this=0x7f5046f57580, j=...) at ../../src/tbb/market.cpp:701
#6  0x00007f5047f4f9f6 in tbb::internal::rml::private_worker::run (this=0x7f5046d79100) at ../../src/tbb/private_server.cpp:270
#7  0x00007f5047f4fc29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#8  0x0000003653407aa1 in start_thread () from /lib64/libpthread.so.0
#9  0x00000036530e8bcd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f502a442700 (LWP 172840)):
#0  0x000000365340b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f504783d53c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/gcc6oldabi/cms/BUILD/slc6_amd64_gcc630/external/gcc/6.3.0-cms/gcc-tags_gcc_6_3_0_release-243837/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f502aa0052b in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/biglib/slc6_amd64_gcc630/pluginSimulation.so
#4  0x00007f5047842ecf in std::execute_native_thread_routine (__p=0x7f502cebf360) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x0000003653407aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000036530e8bcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f5039cbf700 (LWP 172651)):
#0  0x000000365340f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f504517b8f7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f504517c4e5 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f5047842ecf in std::execute_native_thread_routine (__p=0x7f5039e7fc80) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x0000003653407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000036530e8bcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f50474f4c00 (LWP 172439)):
#0  0x00000036530acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000036530acac0 in sleep () from /lib64/libc.so.6
#2  0x00007f504517b6ee in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00000036530e5419 in syscall () from /lib64/libc.so.6
#5  0x00007f5047f4f86c in tbb::internal::futex_wakeup_one (futex=0x7f5046d791ac) at ../../include/tbb/machine/linux_common.h:69
#6  tbb::internal::binary_semaphore::V (this=0x7f5046d791ac) at ../../src/tbb/semaphore.h:217
#7  rml::internal::thread_monitor::notify (this=0x7f5046d791a0) at ../../src/rml/include/../server/thread_monitor.h:243
#8  tbb::internal::rml::private_worker::wake_or_launch (this=0x7f5046d79180) at ../../src/tbb/private_server.cpp:314
#9  tbb::internal::rml::private_server::wake_some (this=<optimized out>, additional_slack=<optimized out>) at ../../src/tbb/private_server.cpp:394
#10 0x00007f5047f558e5 in tbb::internal::arena::enqueue_task (this=0x7f5046f56d00, t=..., prio=<optimized out>, random=...) at ../../src/tbb/arena.cpp:575
#11 0x00007f504934b5d7 in void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetup const&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f504934b8b3 in void edm::endStreamTransitionAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>, edm::RunPrincipal, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> > >(edm::WaitingTaskHolder, edm::Schedule&, unsigned int, edm::RunPrincipal&, edm::IOVSyncValue const&, edm::EventSetup const&, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> >&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f50493afdf5 in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f50493aff4b in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f50493b5e7f in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#16 0x000000000040da34 in main::{lambda()#1}::operator()() const ()
#17 0x000000000040c372 in main ()

[Dr15Jones]

A failure: https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc700/CMSSW_10_1_X_2018-02-04-0000/pyRelValMatrixLogs/run/7.0_Cosmics+Cosmics+DIGICOS+RECOCOS+ALCACOS+HARVESTCOS/step1_Cosmics+Cosmics+DIGICOS+RECOCOS+ALCACOS+HARVESTCOS.log

cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/f6ba31425b8af8631f6126baf5e256da/opt/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Thread 6 (Thread 0x7fd7683ff700 (LWP 14528)):
#0  0x00007fd78d885a3d in poll () from /lib64/libc.so.6
#1  0x00007fd789c2ef57 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007fd789c2f61c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007fd789c306d9 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fd78d7cd1f7 in raise () from /lib64/libc.so.6
#6  0x00007fd78d7ce8e8 in abort () from /lib64/libc.so.6
#7  0x00007fd78d7c6266 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007fd78d7c6312 in __assert_fail () from /lib64/libc.so.6
#9  0x00007fd78ff76d26 in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/libFWCoreConcurrency.so
#10 0x00007fd79031ff69 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007fd7902bfd91 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007fd78eea4466 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fd78b723e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007fd78ee9ddd8 in tbb::internal::arena::process (this=0x7fd78b7d2d00, s=...) at ../../src/tbb/arena.cpp:160
#14 0x00007fd78ee9c8cb in tbb::internal::market::process (this=0x7fd78b7d3580, j=...) at ../../src/tbb/market.cpp:693
#15 0x00007fd78ee98b45 in tbb::internal::rml::private_worker::run (this=0x7fd78b5f5080) at ../../src/tbb/private_server.cpp:270
#16 0x00007fd78ee98d89 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#17 0x00007fd78db63e25 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fd78d89034d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fd7691fe700 (LWP 14527)):
#0  0x00007fd78d8571ad in nanosleep () from /lib64/libc.so.6
#1  0x00007fd78d857044 in sleep () from /lib64/libc.so.6
#2  0x00007fd789c2e831 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fd78d88a7f9 in syscall () from /lib64/libc.so.6
#5  0x00007fd78ee98d62 in tbb::internal::futex_wait (comparand=2, futex=0x7fd78b5f51ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7fd78b5f51ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7fd78b5f51a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7fd78b5f5180) at ../../src/tbb/private_server.cpp:277
#9  0x00007fd78ee98d89 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007fd78db63e25 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fd78d89034d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fd769bff700 (LWP 14526)):
#0  0x00007fd78d8571ad in nanosleep () from /lib64/libc.so.6
#1  0x00007fd78d857044 in sleep () from /lib64/libc.so.6
#2  0x00007fd789c2e831 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fd78d88a7f9 in syscall () from /lib64/libc.so.6
#5  0x00007fd78ee98d62 in tbb::internal::futex_wait (comparand=2, futex=0x7fd78b5f512c) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7fd78b5f512c) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7fd78b5f5120) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7fd78b5f5100) at ../../src/tbb/private_server.cpp:277
#9  0x00007fd78ee98d89 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007fd78db63e25 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fd78d89034d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fd76c9ff700 (LWP 14499)):
#0  0x00007fd78db67945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fd78e1481bc in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc700/external/gcc/7.0.0-omkpbe/gcc-branches_gcc-7-branch-257125/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007fd76d466dfb in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/biglib/slc7_amd64_gcc700/pluginSimulation.so
#4  0x00007fd78e14ddef in std::execute_native_thread_routine (__p=0x7fd76f894280) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x00007fd78db63e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fd78d89034d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fd77d3ff700 (LWP 14435)):
#0  0x00007fd78db6b189 in waitpid () from /lib64/libpthread.so.0
#1  0x00007fd789c2ea37 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007fd789c2f53a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007fd78e14ddef in std::execute_native_thread_routine (__p=0x7fd77dda2cc0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007fd78db63e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fd78d89034d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fd78bf61c80 (LWP 14323)):
#0  0x00007fd78d8571ad in nanosleep () from /lib64/libc.so.6
#1  0x00007fd78d857044 in sleep () from /lib64/libc.so.6
#2  0x00007fd789c2e831 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fd78d874e47 in sched_yield () from /lib64/libc.so.6
#5  0x00007fd78eea3165 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7fd78b7c2600, completion_ref_count=@0x7fd78b7c7828: 2, isolation=0) at ../../src/tbb/custom_scheduler.h:299
#6  0x00007fd78eea459b in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fd78b7c2600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:620
#7  0x00007fd790338628 in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#8  0x00007fd79033873b in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#9  0x0000000000411a36 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#10 0x00007fd79033d299 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x000000000040dad6 in main::{lambda()#1}::operator()() const ()
#12 0x000000000040c3af in main ()

[Dr15Jones]

A failure: https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc700/CMSSW_10_1_X_2018-02-05-1100/pyRelValMatrixLogs/run/106.0_bJpsiX+bJpsiX+DIGI+RECO/step1_bJpsiX+bJpsiX+DIGI+RECO.log

cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/4c403c208f12d99725f04456306d1db2/opt/cmssw/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Thread 6 (Thread 0x7f66453ff700 (LWP 6999)):
#0  0x00007f666cf5b383 in poll () from /lib64/libc.so.6
#1  0x00007f666912d987 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f666912e04c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f666912f109 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f666ceae495 in raise () from /lib64/libc.so.6
#6  0x00007f666ceafc75 in abort () from /lib64/libc.so.6
#7  0x00007f666cea760e in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007f666cea76d0 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f666f5afd16 in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/libFWCoreConcurrency.so
#10 0x00007f666f930f59 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f666f8c9a11 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetup const&, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f666e4dc446 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f666aea7e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007f666e4d5db8 in tbb::internal::arena::process (this=0x7f666af86d00, s=...) at ../../src/tbb/arena.cpp:160
#14 0x00007f666e4d48ab in tbb::internal::market::process (this=0x7f666af87580, j=...) at ../../src/tbb/market.cpp:693
#15 0x00007f666e4d0b25 in tbb::internal::rml::private_worker::run (this=0x7f666adaa080) at ../../src/tbb/private_server.cpp:270
#16 0x00007f666e4d0d69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#17 0x00007f666d217aa1 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f666cf64bcd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f66463fe700 (LWP 6998)):
#0  0x00007f666cf28c4d in nanosleep () from /lib64/libc.so.6
#1  0x00007f666cf28ac0 in sleep () from /lib64/libc.so.6
#2  0x00007f666912d261 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f666cf4b4a7 in sched_yield () from /lib64/libc.so.6
#5  0x00007f666e4db145 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7f666aecfe00, completion_ref_count=@0x7f666aed3f28: 2, isolation=0) at ../../src/tbb/custom_scheduler.h:299
#6  0x00007f666e4dc57b in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f666aecfe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:620
#7  0x00007f666e4d5db8 in tbb::internal::arena::process (this=0x7f666af86d00, s=...) at ../../src/tbb/arena.cpp:160
#8  0x00007f666e4d48ab in tbb::internal::market::process (this=0x7f666af87580, j=...) at ../../src/tbb/market.cpp:693
#9  0x00007f666e4d0b25 in tbb::internal::rml::private_worker::run (this=0x7f666adaa180) at ../../src/tbb/private_server.cpp:270
#10 0x00007f666e4d0d69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#11 0x00007f666d217aa1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f666cf64bcd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f6646dff700 (LWP 6997)):
#0  0x00007f666cf28c4d in nanosleep () from /lib64/libc.so.6
#1  0x00007f666cf28ac0 in sleep () from /lib64/libc.so.6
#2  0x00007f666912d261 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f666cf4b4a7 in sched_yield () from /lib64/libc.so.6
#5  0x00007f666e4db145 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7f666aeefe00, completion_ref_count=@0x7f666aef3f28: 2, isolation=0) at ../../src/tbb/custom_scheduler.h:299
#6  0x00007f666e4dc57b in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f666aeefe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:620
#7  0x00007f666e4d5db8 in tbb::internal::arena::process (this=0x7f666af86d00, s=...) at ../../src/tbb/arena.cpp:160
#8  0x00007f666e4d48ab in tbb::internal::market::process (this=0x7f666af87580, j=...) at ../../src/tbb/market.cpp:693
#9  0x00007f666e4d0b25 in tbb::internal::rml::private_worker::run (this=0x7f666adaa100) at ../../src/tbb/private_server.cpp:270
#10 0x00007f666e4d0d69 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#11 0x00007f666d217aa1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f666cf64bcd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f66499ff700 (LWP 6996)):
#0  0x00007f666d21b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f666d77f18c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc700/external/gcc/7.0.0-okplch/gcc-branches_gcc-7-branch-257125/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f664a6cd01b in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc700/cms/cmssw-patch/CMSSW_10_1_X_2018-02-05-1100/biglib/slc6_amd64_gcc700/pluginSimulation.so
#4  0x00007f666d784dbf in std::execute_native_thread_routine (__p=0x7f664cee4d40) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x00007f666d217aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f666cf64bcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f665cdff700 (LWP 6967)):
#0  0x00007f666d21f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f666912d467 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f666912df6a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f666d784dbf in std::execute_native_thread_routine (__p=0x7f665ce30d20) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007f666d217aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f666cf64bcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f666b65fc00 (LWP 6945)):
#0  0x00007f666cf28c4d in nanosleep () from /lib64/libc.so.6
#1  0x00007f666cf28ac0 in sleep () from /lib64/libc.so.6
#2  0x00007f666912d261 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f666cf4b4a7 in sched_yield () from /lib64/libc.so.6
#5  0x00007f666e4db145 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7f666af76600, completion_ref_count=@0x7f666af7b828: 2, isolation=0) at ../../src/tbb/custom_scheduler.h:299
#6  0x00007f666e4dc57b in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f666af76600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:620
#7  0x00007f666f943218 in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#8  0x00007f666f94332b in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#9  0x0000000000411a96 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#10 0x00007f666f947e89 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02510/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_1_X_2018-02-04-0000/lib/slc6_amd64_gcc700/libFWCoreFramework.so
#11 0x000000000040db36 in main::{lambda()#1}::operator()() const ()
#12 0x000000000040c3ef in main ()

[Dr15Jones]

These appear to happen only on generators steps which have extremely short event processing times because they have a filter which is rejecting the majority of the events.

[Dr15Jones]

I have attempted to reproduce the problem by running the job automatically in gdb. After hundreds of attempts, it never once failed. So the conditions needed to replicate the problem are difficult to setup.

[Dr15Jones]

This happened again https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc630/CMSSW_10_1_X_2018-02-14-0000/pyRelValMatrixLogs/run/1325.5_TTbar_13_reminiaodINPUT+TTbar_13_reminiaodINPUT+REMINIAOD_mc2016+HARVESTDR2_REMINIAOD_mc2016/step2_TTbar_13_reminiaodINPUT+TTbar_13_reminiaodINPUT+REMINIAOD_mc2016+HARVESTDR2_REMINIAOD_mc2016.log

with traceback

Thread 6 (Thread 0x7f9cc37ff700 (LWP 19248)):
#0  0x0000003f8eeacc4d in nanosleep () from /lib64/libc.so.6
#1  0x0000003f8eeacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f9ce1e30f8e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003f8eecf4a7 in sched_yield () from /lib64/libc.so.6
#5  0x00007f9ce4a2114f in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7f9ce3b3fe00, completion_ref_count=@0x7f9ce3b43f28: 2, isolation=0) at ../../src/tbb/custom_scheduler.h:299
#6  0x00007f9ce4a1bbf3 in tbb::internal::arena::process (this=0x7f9ce3c16d00, s=...) at ../../src/tbb/arena.cpp:155
#7  0x00007f9ce4a1a74b in tbb::internal::market::process (this=0x7f9ce3c17580, j=...) at ../../src/tbb/market.cpp:693
#8  0x00007f9ce4a169f6 in tbb::internal::rml::private_worker::run (this=0x7f9ce3a39080) at ../../src/tbb/private_server.cpp:270
#9  0x00007f9ce4a16c29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x0000003f8f207aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003f8eee8bcd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f9cc42be700 (LWP 19247)):
#0  0x0000003f8eeacc4d in nanosleep () from /lib64/libc.so.6
#1  0x0000003f8eeacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f9ce1e30f8e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003f8eecf4a7 in sched_yield () from /lib64/libc.so.6
#5  0x00007f9ce4a2114f in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7f9ce3b77e00, completion_ref_count=@0x7f9ce3b7bf28: 2, isolation=0) at ../../src/tbb/custom_scheduler.h:299
#6  0x00007f9ce4a1bbf3 in tbb::internal::arena::process (this=0x7f9ce3c16d00, s=...) at ../../src/tbb/arena.cpp:155
#7  0x00007f9ce4a1a74b in tbb::internal::market::process (this=0x7f9ce3c17580, j=...) at ../../src/tbb/market.cpp:693
#8  0x00007f9ce4a169f6 in tbb::internal::rml::private_worker::run (this=0x7f9ce3a39180) at ../../src/tbb/private_server.cpp:270
#9  0x00007f9ce4a16c29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x0000003f8f207aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003f8eee8bcd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f9cc4cbf700 (LWP 19246)):
#0  0x0000003f8eedf383 in poll () from /lib64/libc.so.6
#1  0x00007f9ce1e31524 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f9ce1e31e7a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f9ce1e330a6 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000003f8ee32495 in raise () from /lib64/libc.so.6
#6  0x0000003f8ee33c75 in abort () from /lib64/libc.so.6
#7  0x0000003f8ee2b60e in __assert_fail_base () from /lib64/libc.so.6
#8  0x0000003f8ee2b6d0 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f9ce5ae4bce in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/libFWCoreConcurrency.so
#10 0x00007f9ce5e69729 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f9ce5df9985 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetup const&, edm::ServiceToken const&, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f9ce4a2342c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f9ce3b67e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007f9ce4a1bc14 in tbb::internal::arena::process (this=0x7f9ce3c16d00, s=...) at ../../src/tbb/arena.cpp:160
#14 0x00007f9ce4a1a74b in tbb::internal::market::process (this=0x7f9ce3c17580, j=...) at ../../src/tbb/market.cpp:693
#15 0x00007f9ce4a169f6 in tbb::internal::rml::private_worker::run (this=0x7f9ce3a39100) at ../../src/tbb/private_server.cpp:270
#16 0x00007f9ce4a16c29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#17 0x0000003f8f207aa1 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003f8eee8bcd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f9cc6eb8700 (LWP 19173)):
#0  0x0000003f8f20b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9ce450c53c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/gcc6oldabi/cms/BUILD/slc6_amd64_gcc630/external/gcc/6.3.0-cms/gcc-tags_gcc_6_3_0_release-243837/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f9cc7490deb in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/biglib/slc6_amd64_gcc630/pluginSimulation.so
#4  0x00007f9ce4511ecf in std::execute_native_thread_routine (__p=0x7f9ccbf7f400) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x0000003f8f207aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003f8eee8bcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f9cd67bf700 (LWP 18332)):
#0  0x0000003f8f20f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f9ce1e31197 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f9ce1e31d85 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f9ce4511ecf in std::execute_native_thread_routine (__p=0x7f9cd697dcf0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x0000003f8f207aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x0000003f8eee8bcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f9ce41c3c00 (LWP 17397)):
#0  0x0000003f8eeacc4d in nanosleep () from /lib64/libc.so.6
#1  0x0000003f8eeacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f9ce1e30f8e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000003f8eee5419 in syscall () from /lib64/libc.so.6
#5  0x00007f9ce4a1686c in tbb::internal::futex_wakeup_one (futex=0x7f9ce3a391ac) at ../../include/tbb/machine/linux_common.h:69
#6  tbb::internal::binary_semaphore::V (this=0x7f9ce3a391ac) at ../../src/tbb/semaphore.h:217
#7  rml::internal::thread_monitor::notify (this=0x7f9ce3a391a0) at ../../src/rml/include/../server/thread_monitor.h:243
#8  tbb::internal::rml::private_worker::wake_or_launch (this=0x7f9ce3a39180) at ../../src/tbb/private_server.cpp:314
#9  tbb::internal::rml::private_server::wake_some (this=<optimized out>, additional_slack=<optimized out>) at ../../src/tbb/private_server.cpp:394
#10 0x00007f9ce4a1c8e5 in tbb::internal::arena::enqueue_task (this=0x7f9ce3c16d00, t=..., prio=<optimized out>, random=...) at ../../src/tbb/arena.cpp:575
#11 0x00007f9ce5dee7c7 in void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetup const&, edm::ServiceToken const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f9ce5e92d2c in void edm::endStreamsTransitionAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>, edm::RunPrincipal, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> > >(edm::WaitingTaskHolder, edm::Schedule&, unsigned int, edm::RunPrincipal&, edm::IOVSyncValue const&, edm::EventSetup const&, edm::ServiceToken const&, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> >&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f9ce5e859c4 in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f9ce5e85b0b in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f9ce5e8b96f in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-02-14-0000/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#16 0x000000000040da34 in main::{lambda()#1}::operator()() const ()
#17 0x000000000040c372 in main ()

[Dr15Jones]

It happened here: https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc630/CMSSW_10_1_X_2018-03-14-2300/pyRelValMatrixLogs/run/106.0_bJpsiX+bJpsiX+DIGI+RECO/step1_bJpsiX+bJpsiX+DIGI+RECO.log

Thread 6 (Thread 0x7ff553eff700 (LWP 112177)):
#0  0x00000036530acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000036530acac0 in sleep () from /lib64/libc.so.6
#2  0x00007ff574e446be in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00000036530e5419 in syscall () from /lib64/libc.so.6
#5  0x00007ff57819cc02 in tbb::internal::futex_wait (comparand=2, futex=0x7ff576fb90ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7ff576fb90ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7ff576fb90a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7ff576fb9080) at ../../src/tbb/private_server.cpp:277
#9  0x00007ff57819cc29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x0000003653407aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x00000036530e8bcd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7ff55497e700 (LWP 112176)):
#0  0x00000036530df383 in poll () from /lib64/libc.so.6
#1  0x00007ff574e44d04 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007ff574e455fa in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007ff574e466f6 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000003653032495 in raise () from /lib64/libc.so.6
#6  0x0000003653033c75 in abort () from /lib64/libc.so.6
#7  0x000000365302b60e in __assert_fail_base () from /lib64/libc.so.6
#8  0x000000365302b6d0 in __assert_fail () from /lib64/libc.so.6
#9  0x00007ff579264c8e in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02515/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-12-2300/lib/slc6_amd64_gcc630/libFWCoreConcurrency.so
#10 0x00007ff579622759 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#11 0x00007ff5795b25d5 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetup const&, edm::ServiceToken const&, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007ff5781a942c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7ff5770ffe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007ff5781a1c14 in tbb::internal::arena::process (this=0x7ff577196d00, s=...) at ../../src/tbb/arena.cpp:160
#14 0x00007ff5781a074b in tbb::internal::market::process (this=0x7ff577197580, j=...) at ../../src/tbb/market.cpp:693
#15 0x00007ff57819c9f6 in tbb::internal::rml::private_worker::run (this=0x7ff576fb9180) at ../../src/tbb/private_server.cpp:270
#16 0x00007ff57819cc29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#17 0x0000003653407aa1 in start_thread () from /lib64/libpthread.so.0
#18 0x00000036530e8bcd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7ff55537f700 (LWP 112175)):
#0  0x00000036530acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000036530acac0 in sleep () from /lib64/libc.so.6
#2  0x00007ff574e446be in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00000036530e5419 in syscall () from /lib64/libc.so.6
#5  0x00007ff57819cc02 in tbb::internal::futex_wait (comparand=2, futex=0x7ff576fb912c) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7ff576fb912c) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7ff576fb9120) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7ff576fb9100) at ../../src/tbb/private_server.cpp:277
#9  0x00007ff57819cc29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x0000003653407aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x00000036530e8bcd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7ff5576f7700 (LWP 112153)):
#0  0x000000365340b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ff577a882cc in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /mnt/build/davidlt/gcc630/b/BUILD/slc6_amd64_gcc630/external/gcc/6.3.0/gcc-tags_gcc_6_3_0_release-243837/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007ff557cc8d8b in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/biglib/slc6_amd64_gcc630/pluginSimulation.so
#4  0x00007ff577a8dc2f in std::execute_native_thread_routine (__p=0x7ff564c4b8a0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x0000003653407aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x00000036530e8bcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7ff56977f700 (LWP 112119)):
#0  0x000000365340f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007ff574e448c7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007ff574e45525 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007ff577a8dc2f in std::execute_native_thread_routine (__p=0x7ff56a753040) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x0000003653407aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000036530e8bcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7ff577740c20 (LWP 112079)):
#0  0x00000036530acc4d in nanosleep () from /lib64/libc.so.6
#1  0x00000036530acac0 in sleep () from /lib64/libc.so.6
#2  0x00007ff574e446be in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00000036530e5419 in syscall () from /lib64/libc.so.6
#5  0x00007ff57819c86c in tbb::internal::futex_wakeup_one (futex=0x7ff576fb91ac) at ../../include/tbb/machine/linux_common.h:69
#6  tbb::internal::binary_semaphore::V (this=0x7ff576fb91ac) at ../../src/tbb/semaphore.h:217
#7  rml::internal::thread_monitor::notify (this=0x7ff576fb91a0) at ../../src/rml/include/../server/thread_monitor.h:243
#8  tbb::internal::rml::private_worker::wake_or_launch (this=0x7ff576fb9180) at ../../src/tbb/private_server.cpp:314
#9  tbb::internal::rml::private_server::wake_some (this=<optimized out>, additional_slack=<optimized out>) at ../../src/tbb/private_server.cpp:394
#10 0x00007ff5781a5457 in tbb::internal::generic_scheduler::local_spawn (this=0x7ff577186600, first=<optimized out>, next=<optimized out>) at ../../src/tbb/scheduler.cpp:648
#11 0x00007ff5795a792c in void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetup const&, edm::ServiceToken const&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007ff57965029c in void edm::endStreamsTransitionAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>, edm::RunPrincipal, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> > >(edm::WaitingTaskHolder, edm::Schedule&, unsigned int, edm::RunPrincipal&, edm::IOVSyncValue const&, edm::EventSetup const&, edm::ServiceToken const&, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> >&, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#13 0x00007ff5796433e4 in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#14 0x00007ff5796434f3 in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#15 0x00007ff57964362b in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#16 0x00007ff5796490cd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_1_X_2018-03-14-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#17 0x000000000040e782 in main::{lambda()#1}::operator()() const ()
#18 0x000000000040d17a in main ()

Running the job again with the Tracer. The following modules are run at stream end lumi:

g4SimHits
ak4GenJets
VtxSmeared
ak8GenJets
ak4GenJetsNoNu
ak8GenJetsNoNu
genMetCalo
genMetTrue

and items run at stream end Run are

VtxSmeared
ak4GenJets
ak4GenJetsNoNu
ak8GenJets
ak8GenJetsNoNu
g4SimHits
genMetCalo
genMetTrue

which is the same list.

Between those two transitions, the global end Lumi is called which does:

MEtoEDMConverter
genFilterEfficiencyProducer
genXSecAnalyzer
generator
randomEngineStateProducer

as well as writes the Lumi via RAWSIMoutput.

So for a end lumi stream modules to be told to run (which adds an item to the WaitingTaskList) but then, some how, being skipped for synchronization (where we must wait for all stream end lumis to finish) and still not be run by the time we reach stream end run stage seems very difficult.

[Dr15Jones]

It happened here: https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc630/CMSSW_10_1_X_2018-03-11-0000/pyRelValMatrixLogs/run/136.7803_RunHLTPhy2017B_RAWAOD+RunHLTPhy2017B_RAWAOD+DQMHLTonRAWAOD_2017/step2_RunHLTPhy2017B_RAWAOD+RunHLTPhy2017B_RAWAOD+DQMHLTonRAWAOD_2017.log

Thread 10 (Thread 0x7ff4a3a3f700 (LWP 31367)):
#0  0x00007ff4cfdf91ad in nanosleep () from /lib64/libc.so.6
#1  0x00007ff4cfdf9044 in sleep () from /lib64/libc.so.6
#2  0x00007ff4c53551ee in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ff4cfe16e47 in sched_yield () from /lib64/libc.so.6
#5  0x00007ff4d144e17f in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7ff4cded7e00, completion_ref_count=@0x7ff4cdedbf28: 2, isolation=0) at ../../src/tbb/custom_scheduler.h:299
#6  0x00007ff4d1448c23 in tbb::internal::arena::process (this=0x7ff4cdf96d00, s=...) at ../../src/tbb/arena.cpp:155
#7  0x00007ff4d144777b in tbb::internal::market::process (this=0x7ff4cdf97580, j=...) at ../../src/tbb/market.cpp:693
#8  0x00007ff4d1443a26 in tbb::internal::rml::private_worker::run (this=0x7ff4cddb9080) at ../../src/tbb/private_server.cpp:270
#9  0x00007ff4d1443c59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007ff4d0105e25 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ff4cfe3234d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7ff4a44be700 (LWP 31366)):
#0  0x00007ff4cfdf91ad in nanosleep () from /lib64/libc.so.6
#1  0x00007ff4cfdf9044 in sleep () from /lib64/libc.so.6
#2  0x00007ff4c53551ee in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ff4cfe2c7f9 in syscall () from /lib64/libc.so.6
#5  0x00007ff4d1443c32 in tbb::internal::futex_wait (comparand=2, futex=0x7ff4cddb91ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7ff4cddb91ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7ff4cddb91a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7ff4cddb9180) at ../../src/tbb/private_server.cpp:277
#9  0x00007ff4d1443c59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007ff4d0105e25 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ff4cfe3234d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7ff4a4ebf700 (LWP 31365)):
#0  0x00007ff4cfdf91ad in nanosleep () from /lib64/libc.so.6
#1  0x00007ff4cfdf9044 in sleep () from /lib64/libc.so.6
#2  0x00007ff4c53551ee in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ff4cfe2c7f9 in syscall () from /lib64/libc.so.6
#5  0x00007ff4d1443c32 in tbb::internal::futex_wait (comparand=2, futex=0x7ff4cddb912c) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7ff4cddb912c) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7ff4cddb9120) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7ff4cddb9100) at ../../src/tbb/private_server.cpp:277
#9  0x00007ff4d1443c59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007ff4d0105e25 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ff4cfe3234d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7ff4af069700 (LWP 31311)):
#0  0x00007ff4cfe2c7f9 in syscall () from /lib64/libc.so.6
#1  0x00007ff4c9d8b89f in XrdSys::LinuxSemaphore::Wait (this=0x7ff4b60b7988) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7ff4b60e7f58) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7ff4b60e7f40) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007ff4c9d8ba59 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/XrdCl/XrdClJobManager.cc:33
#5  0x00007ff4d0105e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ff4cfe3234d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7ff4afa6a700 (LWP 31310)):
#0  0x00007ff4cfe2c7f9 in syscall () from /lib64/libc.so.6
#1  0x00007ff4c9d8b89f in XrdSys::LinuxSemaphore::Wait (this=0x7ff4b60b7988) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7ff4b60e7f58) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7ff4b60e7f40) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007ff4c9d8ba59 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/XrdCl/XrdClJobManager.cc:33
#5  0x00007ff4d0105e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ff4cfe3234d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7ff4b046b700 (LWP 31309)):
#0  0x00007ff4cfe2c7f9 in syscall () from /lib64/libc.so.6
#1  0x00007ff4c9d8b89f in XrdSys::LinuxSemaphore::Wait (this=0x7ff4b60b7988) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7ff4b60e7f58) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7ff4b60e7f40) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007ff4c9d8ba59 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/XrdCl/XrdClJobManager.cc:33
#5  0x00007ff4d0105e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ff4cfe3234d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7ff4b0e6c700 (LWP 31308)):
#0  0x00007ff4d010ce4d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007ff4cacd9753 in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/XrdSys/XrdSysTimer.cc:239
#2  0x00007ff4c9d30627 in XrdCl::TaskManager::RunTasks (this=0x7ff4b608b440) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/XrdCl/XrdClTaskManager.cc:244
#3  0x00007ff4c9d30779 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/XrdCl/XrdClTaskManager.cc:37
#4  0x00007ff4d0105e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ff4cfe3234d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7ff4b186d700 (LWP 31307)):
#0  0x00007ff4cfe32923 in epoll_wait () from /lib64/libc.so.6
#1  0x00007ff4cacde804 in XrdSys::IOEvents::PollE::Begin (this=0x7ff4b60e7e60, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007ff4cacdae75 in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffda28bacd0) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/XrdSys/XrdSysIOEvents.cc:131
#3  0x00007ff4cacd8f78 in XrdSysThread_Xeq (myargs=0x7ff4b1d20c40) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc630/external/xrootd/4.6.1-omkpbe3/xrootd-4.6.1/src/XrdSys/XrdSysPthread.cc:86
#4  0x00007ff4d0105e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ff4cfe3234d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7ff4b793f700 (LWP 31301)):
#0  0x00007ff4d010d189 in waitpid () from /lib64/libpthread.so.0
#1  0x00007ff4c53553f7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007ff4c5356055 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007ff4d06eec3f in std::execute_native_thread_routine (__p=0x7ff4c5ebba20) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007ff4d0105e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ff4cfe3234d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7ff4ce4fdc80 (LWP 31254)):
#0  0x00007ff4cfe27a3d in poll () from /lib64/libc.so.6
#1  0x00007ff4c5355834 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007ff4c535612a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007ff4c5357226 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007ff4cfd6f1f7 in raise () from /lib64/libc.so.6
#6  0x00007ff4cfd708e8 in abort () from /lib64/libc.so.6
#7  0x00007ff4cfd68266 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007ff4cfd68312 in __assert_fail () from /lib64/libc.so.6
#9  0x00007ff4d2509cae in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/libFWCoreConcurrency.so
#10 0x00007ff4d28a5d69 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#11 0x00007ff4d2855395 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetup const&, edm::ServiceToken const&, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007ff4d145045c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7ff4cdf86600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007ff4d28c25af in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007ff4d28c26a3 in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007ff4d28c27db in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#16 0x00007ff4d28c827d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_1_X_2018-03-11-0000/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#17 0x000000000040e732 in main::{lambda()#1}::operator()() const ()
#18 0x000000000040d14a in main ()

[Dr15Jones]

The problem re-occurred in CMSSW_10_2_X_2018-05-09-1100 workflow 7.0

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc7_amd64_gcc630/CMSSW_10_2_X_2018-05-09-1100/pyRelValMatrixLogs/run/7.0_Cosmics+Cosmics+DIGICOS+RECOCOS+ALCACOS+HARVESTCOS/step1_Cosmics+Cosmics+DIGICOS+RECOCOS+ALCACOS+HARVESTCOS.log

cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/fe7678ab15867201e450c252338549a1/opt/cmssw/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Thread 6 (Thread 0x7fe42273f700 (LWP 60351)):
#0  0x00007fe447e801ad in nanosleep () from /lib64/libc.so.6
#1  0x00007fe447e80044 in sleep () from /lib64/libc.so.6
#2  0x00007fe4444d716e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe44ab0d705 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fe44ab0dfdf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007fe44ab128de in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007fe44ab19f5a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007fe426dfb949 in RunAction::~RunAction() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/biglib/slc7_amd64_gcc630/pluginSimulation.so
#9  0x00007fe426e053c5 in RunManagerMTWorker::terminateRun() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/biglib/slc7_amd64_gcc630/pluginSimulation.so
#10 0x00007fe426dc6974 in OscarMTProducer::endRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/biglib/slc7_amd64_gcc630/pluginSimulation.so
#11 0x00007fe44aa63f7b in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007fe44aa156d0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#13 0x00007fe44a958157 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007fe44a95839c in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007fe44a958574 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#16 0x00007fe44a95863d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#17 0x00007fe4494d745c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe445f5fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#18 0x00007fe4494cfc44 in tbb::internal::arena::process (this=0x7fe446016d00, s=...) at ../../src/tbb/arena.cpp:160
#19 0x00007fe4494ce77b in tbb::internal::market::process (this=0x7fe446017580, j=...) at ../../src/tbb/market.cpp:693
#20 0x00007fe4494caa26 in tbb::internal::rml::private_worker::run (this=0x7fe445e39080) at ../../src/tbb/private_server.cpp:270
#21 0x00007fe4494cac59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#22 0x00007fe44818be25 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fe447eb934d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fe4231be700 (LWP 60350)):
#0  0x00007fe447e801ad in nanosleep () from /lib64/libc.so.6
#1  0x00007fe447e80044 in sleep () from /lib64/libc.so.6
#2  0x00007fe4444d716e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe44ab0d705 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fe44ab0dfdf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007fe44ab128de in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007fe44ab19f5a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007fe426dfb949 in RunAction::~RunAction() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/biglib/slc7_amd64_gcc630/pluginSimulation.so
#9  0x00007fe426e053c5 in RunManagerMTWorker::terminateRun() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/biglib/slc7_amd64_gcc630/pluginSimulation.so
#10 0x00007fe426dc6974 in OscarMTProducer::endRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/biglib/slc7_amd64_gcc630/pluginSimulation.so
#11 0x00007fe44aa63f7b in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007fe44aa156d0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#13 0x00007fe44a958157 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007fe44a95839c in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007fe44a958574 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#16 0x00007fe44a95863d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#17 0x00007fe4494d745c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe445f6fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#18 0x00007fe4494cfc44 in tbb::internal::arena::process (this=0x7fe446016d00, s=...) at ../../src/tbb/arena.cpp:160
#19 0x00007fe4494ce77b in tbb::internal::market::process (this=0x7fe446017580, j=...) at ../../src/tbb/market.cpp:693
#20 0x00007fe4494caa26 in tbb::internal::rml::private_worker::run (this=0x7fe445e39180) at ../../src/tbb/private_server.cpp:270
#21 0x00007fe4494cac59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#22 0x00007fe44818be25 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fe447eb934d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fe423bbf700 (LWP 60349)):
#0  0x00007fe447e801ad in nanosleep () from /lib64/libc.so.6
#1  0x00007fe447e80044 in sleep () from /lib64/libc.so.6
#2  0x00007fe4444d716e in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe44ab0dc80 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fe44ab0dfdf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007fe44ab128de in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007fe44ab19f5a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007fe426dc6974 in OscarMTProducer::endRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/biglib/slc7_amd64_gcc630/pluginSimulation.so
#9  0x00007fe44aa63f7b in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#10 0x00007fe44aa156d0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&, edm::EventSetup const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#11 0x00007fe44a958157 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007fe44a95839c in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#13 0x00007fe44a958574 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007fe44a95863d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetup const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007fe4494d745c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe445f4fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#16 0x00007fe4494cfc44 in tbb::internal::arena::process (this=0x7fe446016d00, s=...) at ../../src/tbb/arena.cpp:160
#17 0x00007fe4494ce77b in tbb::internal::market::process (this=0x7fe446017580, j=...) at ../../src/tbb/market.cpp:693
#18 0x00007fe4494caa26 in tbb::internal::rml::private_worker::run (this=0x7fe445e39100) at ../../src/tbb/private_server.cpp:270
#19 0x00007fe4494cac59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#20 0x00007fe44818be25 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fe447eb934d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fe42680c700 (LWP 60345)):
#0  0x00007fe44818f945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe44876f2dc in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/davidlt/b/BUILD/slc7_amd64_gcc630/external/gcc/6.3.0/gcc-tags_gcc_6_3_0_release-243837/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007fe426df960b in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/biglib/slc7_amd64_gcc630/pluginSimulation.so
#4  0x00007fe448774c3f in std::execute_native_thread_routine (__p=0x7fe432835080) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x00007fe44818be25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fe447eb934d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fe436a3f700 (LWP 60328)):
#0  0x00007fe448193189 in waitpid () from /lib64/libpthread.so.0
#1  0x00007fe4444d7377 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007fe4444d7fd5 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007fe448774c3f in std::execute_native_thread_routine (__p=0x7fe445a2ea80) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007fe44818be25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe447eb934d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fe446582cc0 (LWP 60291)):
#0  0x00007fe447eaea3d in poll () from /lib64/libc.so.6
#1  0x00007fe4444d77b4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007fe4444d80aa in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007fe4444d91a6 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fe447df61f7 in raise () from /lib64/libc.so.6
#6  0x00007fe447df78e8 in abort () from /lib64/libc.so.6
#7  0x00007fe447def266 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007fe447def312 in __assert_fail () from /lib64/libc.so.6
#9  0x00007fe44a61ccae in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreConcurrency.so
#10 0x00007fe44a9d40d9 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#11 0x00007fe44a95bf95 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetup const&, edm::ServiceToken const&, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007fe4494d745c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe446006600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007fe44a9eda6f in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007fe44a9edb63 in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007fe44a9edc9b in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#16 0x00007fe44a9f36cd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02523/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-09-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#17 0x000000000040e732 in main::{lambda()#1}::operator()() const ()
#18 0x000000000040d14a in main ()

Current Modules:

Module: none (crashed)
Module: OscarMTProducer:g4SimHits
Module: OscarMTProducer:g4SimHits
Module: OscarMTProducer:g4SimHits

A fatal system signal has occurred: abort signal

[Dr15Jones]

The problem happened again in CMSSW_10_2_X_2018-05-26-1100 for architecture slc7_amd64_gcc630 workflow 7.0 step 1.

cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/9175cf5199fd36ddbb6c90e06076fc10/opt/cmssw/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-25-2300/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Thread 6 (Thread 0x7f9c2a51a700 (LWP 50084)):
#0  0x00007f9c53c291ad in nanosleep () from /lib64/libc.so.6
#1  0x00007f9c53c29044 in sleep () from /lib64/libc.so.6
#2  0x00007f9c4f62ffee in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-25-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f9c53c5c7f9 in syscall () from /lib64/libc.so.6
#5  0x00007f9c55273c32 in tbb::internal::futex_wait (comparand=2, futex=0x7f9c5166b0ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f9c5166b0ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7f9c5166b0a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f9c5166b080) at ../../src/tbb/private_server.cpp:277
#9  0x00007f9c55273c59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007f9c53f34e25 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f9c53c6234d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f9c2b51b700 (LWP 50083)):
#0  0x00007f9c53c57a3d in poll () from /lib64/libc.so.6
#1  0x00007f9c4f630634 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-25-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f9c4f630f2a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-25-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f9c4f632026 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-25-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f9c53b9f1f7 in raise () from /lib64/libc.so.6
#6  0x00007f9c53ba08e8 in abort () from /lib64/libc.so.6
#7  0x00007f9c53b98266 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007f9c53b98312 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f9c565f0cae in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-25-2300/lib/slc7_amd64_gcc630/libFWCoreConcurrency.so
#10 0x00007f9c56988209 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_X_2018-05-26-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f9c56944345 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_X_2018-05-26-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f9c5528045c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f9c5179fe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007f9c55278c44 in tbb::internal::arena::process (this=0x7f9c51846d00, s=...) at ../../src/tbb/arena.cpp:160
#14 0x00007f9c5527777b in tbb::internal::market::process (this=0x7f9c51847580, j=...) at ../../src/tbb/market.cpp:693
#15 0x00007f9c55273a26 in tbb::internal::rml::private_worker::run (this=0x7f9c5166b180) at ../../src/tbb/private_server.cpp:270
#16 0x00007f9c55273c59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#17 0x00007f9c53f34e25 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f9c53c6234d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f9c2bf1c700 (LWP 50082)):
#0  0x00007f9c53c291ad in nanosleep () from /lib64/libc.so.6
#1  0x00007f9c53c29044 in sleep () from /lib64/libc.so.6
#2  0x00007f9c4f62ffee in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-25-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f9c53c5c7f9 in syscall () from /lib64/libc.so.6
#5  0x00007f9c55273c32 in tbb::internal::futex_wait (comparand=2, futex=0x7f9c5166b12c) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f9c5166b12c) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7f9c5166b120) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f9c5166b100) at ../../src/tbb/private_server.cpp:277
#9  0x00007f9c55273c59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007f9c53f34e25 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f9c53c6234d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f9c2f5a0700 (LWP 50081)):
#0  0x00007f9c53f38945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9c545182dc in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/davidlt/b/BUILD/slc7_amd64_gcc630/external/gcc/6.3.0/gcc-tags_gcc_6_3_0_release-243837/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f9c2fb92c9b in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&amp)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_X_2018-05-26-1100/biglib/slc7_amd64_gcc630/pluginSimulation.so
#4  0x00007f9c5451dc3f in std::execute_native_thread_routine (__p=0x7f9c3e9c5320) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x00007f9c53f34e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f9c53c6234d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f9c418f2700 (LWP 50067)):
#0  0x00007f9c53f3c189 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f9c4f6301f7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-25-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f9c4f630e55 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-25-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f9c5451dc3f in std::execute_native_thread_routine (__p=0x7f9c51082a80) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007f9c53f34e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f9c53c6234d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f9c5232c440 (LWP 50038)):
#0  0x00007f9c53c291ad in nanosleep () from /lib64/libc.so.6
#1  0x00007f9c53c29044 in sleep () from /lib64/libc.so.6
#2  0x00007f9c4f62ffee in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-05-25-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f9c53c5c7f9 in syscall () from /lib64/libc.so.6
#5  0x00007f9c5527389c in tbb::internal::futex_wakeup_one (futex=0x7f9c5166b1ac) at ../../include/tbb/machine/linux_common.h:69
#6  tbb::internal::binary_semaphore::V (this=0x7f9c5166b1ac) at ../../src/tbb/semaphore.h:217
#7  rml::internal::thread_monitor::notify (this=0x7f9c5166b1a0) at ../../src/rml/include/../server/thread_monitor.h:243
#8  tbb::internal::rml::private_worker::wake_or_launch (this=0x7f9c5166b180) at ../../src/tbb/private_server.cpp:314
#9  tbb::internal::rml::private_server::wake_some (this=<optimized out>, additional_slack=<optimized out>) at ../../src/tbb/private_server.cpp:394
#10 0x00007f9c55279915 in tbb::internal::arena::enqueue_task (this=0x7f9c51846d00, t=..., prio=<optimized out>, random=...) at ../../src/tbb/arena.cpp:575
#11 0x00007f9c569397e7 in void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_X_2018-05-26-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f9c569b423c in void edm::endStreamsTransitionAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>, edm::RunPrincipal, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> > >(edm::WaitingTaskHolder, edm::Schedule&amp, unsigned int, edm::RunPrincipal&amp, edm::IOVSyncValue const&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> >&amp, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_X_2018-05-26-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f9c569a73f4 in edm::EventProcessor::endRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_X_2018-05-26-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f9c569a7503 in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_X_2018-05-26-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f9c569a763b in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_X_2018-05-26-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#16 0x00007f9c569ad06d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02525/slc7_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_X_2018-05-26-1100/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#17 0x000000000040e712 in main::{lambda()#1}::operator()() const ()
#18 0x000000000040d12a in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: none

What is interesting about this is the main thread appears to still be looping through the different streams to start them up for the endRun. It also looks like only one stream has started to be processed by a TBB thread.

[Dr15Jones]

I've had a new thought about this. What if this is being caused by an exception going off during the end Stream LuminosityBlock processing? Could that lead to a WaitingTaskList not being called back becuase we decrement a count too soon?

[Dr15Jones]

The problem happened again in CMSSW_10_2_ROOT6_X_2018-05-30-2300 for slc6_amd64_gcc630 in workflow 7.4 step1

cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5c971af74d1656b108c3327466695a5b/opt/cmssw/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_ROOT6_X_2018-05-28-2300/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Thread 6 (Thread 0x7f3fbc6ff700 (LWP 17405)):
#0  0x00007f3fe70bbc4d in nanosleep () from /lib64/libc.so.6
#1  0x00007f3fe70bbac0 in sleep () from /lib64/libc.so.6
#2  0x00007f3fe28c4fae in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_ROOT6_X_2018-05-28-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f3fe70f4419 in syscall () from /lib64/libc.so.6
#5  0x00007f3fe866cc02 in tbb::internal::futex_wait (comparand=2, futex=0x7f3fe4aa30ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f3fe4aa30ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7f3fe4aa30a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f3fe4aa3080) at ../../src/tbb/private_server.cpp:277
#9  0x00007f3fe866cc29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007f3fe73aaaa1 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f3fe70f7bcd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f3fbd500700 (LWP 17404)):
#0  0x00007f3fe70bbc4d in nanosleep () from /lib64/libc.so.6
#1  0x00007f3fe70bbac0 in sleep () from /lib64/libc.so.6
#2  0x00007f3fe28c4fae in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_ROOT6_X_2018-05-28-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f3fe70f4419 in syscall () from /lib64/libc.so.6
#5  0x00007f3fe866cc02 in tbb::internal::futex_wait (comparand=2, futex=0x7f3fe4aa31ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f3fe4aa31ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7f3fe4aa31a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f3fe4aa3180) at ../../src/tbb/private_server.cpp:277
#9  0x00007f3fe866cc29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007f3fe73aaaa1 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f3fe70f7bcd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f3fbdf01700 (LWP 17403)):
#0  0x00007f3fe70bbc4d in nanosleep () from /lib64/libc.so.6
#1  0x00007f3fe70bbac0 in sleep () from /lib64/libc.so.6
#2  0x00007f3fe28c4fae in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_ROOT6_X_2018-05-28-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f3fe70f4419 in syscall () from /lib64/libc.so.6
#5  0x00007f3fe866cc02 in tbb::internal::futex_wait (comparand=2, futex=0x7f3fe4aa312c) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f3fe4aa312c) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7f3fe4aa3120) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f3fe4aa3100) at ../../src/tbb/private_server.cpp:277
#9  0x00007f3fe866cc29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007f3fe73aaaa1 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f3fe70f7bcd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f3fc1379700 (LWP 17328)):
#0  0x00007f3fe73ae68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f3fe79112cc in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /mnt/build/davidlt/gcc630/b/BUILD/slc6_amd64_gcc630/external/gcc/6.3.0/gcc-tags_gcc_6_3_0_release-243837/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f3fc19717bb in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&amp)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_ROOT6_X_2018-05-30-2300/biglib/slc6_amd64_gcc630/pluginSimulation.so
#4  0x00007f3fe7916c2f in std::execute_native_thread_routine (__p=0x7f3fcf6a08c0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x00007f3fe73aaaa1 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f3fe70f7bcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f3fd107a700 (LWP 17114)):
#0  0x00007f3fe73b237d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f3fe28c51b7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_ROOT6_X_2018-05-28-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f3fe28c5e15 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_ROOT6_X_2018-05-28-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f3fe7916c2f in std::execute_native_thread_routine (__p=0x7f3fe3baab50) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007f3fe73aaaa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f3fe70f7bcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f3fe575f3e0 (LWP 16916)):
#0  0x00007f3fe70ee383 in poll () from /lib64/libc.so.6
#1  0x00007f3fe28c55f4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_ROOT6_X_2018-05-28-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f3fe28c5eea in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_ROOT6_X_2018-05-28-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f3fe28c6fe6 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_ROOT6_X_2018-05-28-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f3fe7041495 in raise () from /lib64/libc.so.6
#6  0x00007f3fe7042c75 in abort () from /lib64/libc.so.6
#7  0x00007f3fe703a60e in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007f3fe703a6d0 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f3fe9fb3c8e in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_ROOT6_X_2018-05-28-2300/lib/slc6_amd64_gcc630/libFWCoreConcurrency.so
#10 0x00007f3fea352009 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_ROOT6_X_2018-05-30-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f3fea2fa335 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_ROOT6_X_2018-05-30-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f3fe867942c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f3fe4c6e600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007f3fea37606f in edm::EventProcessor::endRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_ROOT6_X_2018-05-30-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f3fea376163 in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_ROOT6_X_2018-05-30-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f3fea37629b in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_ROOT6_X_2018-05-30-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#16 0x00007f3fea37bccd in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02526/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_ROOT6_X_2018-05-30-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#17 0x000000000040e762 in main::{lambda()#1}::operator()() const ()
#18 0x000000000040d15a in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: none

[Dr15Jones]

This happened again in https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/raw/slc6_amd64_gcc630/CMSSW_10_2_X_2018-06-13-2300/pyRelValMatrixLogs/run/136.7803_RunHLTPhy2017B_RAWAOD+RunHLTPhy2017B_RAWAOD+DQMHLTonRAWAOD_2017+HARVESTDQMHLTonAOD_2017/step2_RunHLTPhy2017B_RAWAOD+RunHLTPhy2017B_RAWAOD+DQMHLTonRAWAOD_2017+HARVESTDQMHLTonAOD_2017.log

Begin processing the 100th record. Run 297557, Event 196136196, LumiSection 123 on stream 2 at 14-Jun-2018 08:16:08.366 CEST
cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/d69cb5cb2017703f9eb49357dd146633/opt/cmssw/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Thread 10 (Thread 0x7f268e7ff700 (LWP 8004)):
#0  0x000000330caacc4d in nanosleep () from /lib64/libc.so.6
#1  0x000000330caacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f26bb11dfae in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ffcfd9e8a11 in clock_gettime ()
#5  0x00007f26bffbfea6 in clock_gettime () from /lib64/librt.so.1
#6  0x00007f26bb07bc11 in boost::chrono::thread_clock::now() () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/external/slc6_amd64_gcc630/lib/libboost_chrono.so.1.63.0
#7  0x00007f26b816cd63 in FastTimerService::Measurement::measure_and_accumulate(FastTimerService::AtomicResources&amp) () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libHLTriggerTimer.so
#8  0x00007f26c1caacf3 in void edm::signalslot::Signal<void (edm::StreamContext const&amp, edm::ModuleCallingContext const&amp)>::emit<edm::StreamContext const&amp, edm::ModuleCallingContext const&amp>(edm::StreamContext const&amp, edm::ModuleCallingContext const&amp) const () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#9  0x00007f26c1cb597c in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#10 0x00007f26c1cb5bdc in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f26c1cb5db4 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f26c1cb5e7d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f26c05f042c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f26bef1be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#14 0x00007f26c05e8c14 in tbb::internal::arena::process (this=0x7f26befc2d00, s=...) at ../../src/tbb/arena.cpp:160
#15 0x00007f26c05e774b in tbb::internal::market::process (this=0x7f26befc3580, j=...) at ../../src/tbb/market.cpp:693
#16 0x00007f26c05e39f6 in tbb::internal::rml::private_worker::run (this=0x7f26bede8080) at ../../src/tbb/private_server.cpp:270
#17 0x00007f26c05e3c29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#18 0x000000330ce07aa1 in start_thread () from /lib64/libpthread.so.0
#19 0x000000330cae8bcd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f268f7df700 (LWP 8003)):
#0  0x000000330caacc4d in nanosleep () from /lib64/libc.so.6
#1  0x000000330caacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f26bb11dfae in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  _dl_fixup (l=0x7f26c207a000, reloc_arg=<optimized out>) at ../elf/dl-runtime.c:90
#5  0x00007f26c1e71825 in _dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:41
#6  0x00007f26c1cb5bdc in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#7  0x00007f26c1cb5db4 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#8  0x00007f26c1cb5e7d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#9  0x00007f26c05f042c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f26bef13e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#10 0x00007f26c05e8c14 in tbb::internal::arena::process (this=0x7f26befc2d00, s=...) at ../../src/tbb/arena.cpp:160
#11 0x00007f26c05e774b in tbb::internal::market::process (this=0x7f26befc3580, j=...) at ../../src/tbb/market.cpp:693
#12 0x00007f26c05e39f6 in tbb::internal::rml::private_worker::run (this=0x7f26bede8180) at ../../src/tbb/private_server.cpp:270
#13 0x00007f26c05e3c29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#14 0x000000330ce07aa1 in start_thread () from /lib64/libpthread.so.0
#15 0x000000330cae8bcd in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f26901e0700 (LWP 8002)):
#0  0x000000330caacc4d in nanosleep () from /lib64/libc.so.6
#1  0x000000330caacac0 in sleep () from /lib64/libc.so.6
#2  0x00007f26bb11dfae in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000000330cae5419 in syscall () from /lib64/libc.so.6
#5  0x00007f26c05e3c02 in tbb::internal::futex_wait (comparand=2, futex=0x7f26bede812c) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f26bede812c) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7f26bede8120) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f26bede8100) at ../../src/tbb/private_server.cpp:277
#9  0x00007f26c05e3c29 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x000000330ce07aa1 in start_thread () from /lib64/libpthread.so.0
#11 0x000000330cae8bcd in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f269c73d700 (LWP 7995)):
#0  0x000000330cae5419 in syscall () from /lib64/libc.so.6
#1  0x00007f26bb83618f in XrdSys::LinuxSemaphore::Wait (this=0x7f26ba6b7e90) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f26b9114318) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f26b9114300) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f26bb836349 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/XrdCl/XrdClJobManager.cc:33
#5  0x000000330ce07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x000000330cae8bcd in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f269d13e700 (LWP 7994)):
#0  0x000000330cae5419 in syscall () from /lib64/libc.so.6
#1  0x00007f26bb83618f in XrdSys::LinuxSemaphore::Wait (this=0x7f26ba6b7e90) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f26b9114318) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f26b9114300) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f26bb836349 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/XrdCl/XrdClJobManager.cc:33
#5  0x000000330ce07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x000000330cae8bcd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f269db3f700 (LWP 7993)):
#0  0x000000330cae5419 in syscall () from /lib64/libc.so.6
#1  0x00007f26bb83618f in XrdSys::LinuxSemaphore::Wait (this=0x7f26ba6b7e90) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/./XrdSys/XrdSysLinuxSemaphore.hh:161
#2  XrdCl::SyncQueue<XrdCl::JobManager::JobHelper>::Get (this=0x7f26b9114318) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/./XrdCl/XrdClSyncQueue.hh:67
#3  XrdCl::JobManager::RunJobs (this=0x7f26b9114300) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/XrdCl/XrdClJobManager.cc:146
#4  0x00007f26bb836349 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/XrdCl/XrdClJobManager.cc:33
#5  0x000000330ce07aa1 in start_thread () from /lib64/libpthread.so.0
#6  0x000000330cae8bcd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f269e540700 (LWP 7992)):
#0  0x000000330ce0f00d in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f26bb38bef3 in XrdSysTimer::Wait (mills=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/XrdSys/XrdSysTimer.cc:239
#2  0x00007f26bb7d8e97 in XrdCl::TaskManager::RunTasks (this=0x7f26b8acca80) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/XrdCl/XrdClTaskManager.cc:244
#3  0x00007f26bb7d8fe9 in RunRunnerThread (arg=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/XrdCl/XrdClTaskManager.cc:37
#4  0x000000330ce07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x000000330cae8bcd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f269ef41700 (LWP 7991)):
#0  0x000000330cae91c3 in epoll_wait () from /lib64/libc.so.6
#1  0x00007f26bb390fc4 in XrdSys::IOEvents::PollE::Begin (this=0x7f26b9114220, syncsem=<optimized out>, retcode=<optimized out>, eTxt=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/./XrdSys/XrdSysIOEventsPollE.icc:213
#2  0x00007f26bb38d625 in XrdSys::IOEvents::BootStrap::Start (parg=0x7ffcfd9019d0) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/XrdSys/XrdSysIOEvents.cc:131
#3  0x00007f26bb38b718 in XrdSysThread_Xeq (myargs=0x7f26ba800360) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc6_amd64_gcc630/external/xrootd/4.8.3-gnimlf/xrootd-4.8.3/src/XrdSys/XrdSysPthread.cc:86
#4  0x000000330ce07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x000000330cae8bcd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f26a3fc4700 (LWP 7935)):
#0  0x000000330ce0f37d in waitpid () from /lib64/libpthread.so.0
#1  0x00007f26bb11e1b7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f26bb11ee15 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f26bfed6c2f in std::execute_native_thread_routine (__p=0x7f26bbbeb630) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x000000330ce07aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x000000330cae8bcd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f26bfb883e0 (LWP 7849)):
#0  0x000000330cadf383 in poll () from /lib64/libc.so.6
#1  0x00007f26bb11e5f4 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f26bb11eeea in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f26bb11ffe6 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000000330ca32495 in raise () from /lib64/libc.so.6
#6  0x000000330ca33c75 in abort () from /lib64/libc.so.6
#7  0x000000330ca2b60e in __assert_fail_base () from /lib64/libc.so.6
#8  0x000000330ca2b6d0 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f26c1974cfe in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreConcurrency.so
#10 0x00007f26c1d0ad79 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f26c1cb97d5 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f26c05f042c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f26befb2600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007f26c1d22bdf in edm::EventProcessor::endRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f26c1d22cd3 in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f26c1d22e0b in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#16 0x00007f26c1d2880d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02528/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-06-13-2300/lib/slc6_amd64_gcc630/libFWCoreFramework.so
#17 0x000000000040e762 in main::{lambda()#1}::operator()() const ()
#18 0x000000000040d15a in main ()

Current Modules:

Module: none (crashed)
Module: TriggerSummaryProducerRAW:hltTriggerSummaryRAW
Module: none
Module: VersionedGsfElectronIdProducer:egmGsfElectronIDsForDQM

A fatal system signal has occurred: abort signal

[Dr15Jones]

This happened again in https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc630/CMSSW_10_2_X_2018-07-13-2300/pyRelValMatrixLogs/run/106.0_bJpsiX+bJpsiX+DIGI+RECO/step1_bJpsiX+bJpsiX+DIGI+RECO.log#/

Begin processing the 10th record. Run 1, Event 10, LumiSection 1 on stream 2 at 14-Jul-2018 06:10:07.946 CEST

 *-------  PYTHIA Event and Cross Section Statistics  -------------------------------------------------------------*
[stuff cut]
 *-------  End PYTHIA Event and Cross Section Statistics ----------------------------------------------------------*

 *-------  PYTHIA Error and Warning Messages Statistics  ----------------------------------------------------------* 
[stuff cut]
 *-------  End PYTHIA Error and Warning Messages Statistics  ------------------------------------------------------* 
cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/d0aa8fdff0eb4a10f70eb1684d0d5563/opt/cmssw/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Thread 6 (Thread 0x7f28465ff700 (LWP 10108)):
#0  0x00007f2871a93a3d in poll () from /lib64/libc.so.6
#1  0x00007f286d861124 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f286d861a1a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f286d862b16 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f28719db1f7 in raise () from /lib64/libc.so.6
#6  0x00007f28719dc8e8 in abort () from /lib64/libc.so.6
#7  0x00007f28719d4266 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007f28719d4312 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f2874446d1e in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreConcurrency.so
#10 0x00007f28747e4ad9 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f2874789ed5 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f28730bc45c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f286f50be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#13 0x00007f28730b4c44 in tbb::internal::arena::process (this=0x7f286f5d2d00, s=...) at ../../src/tbb/arena.cpp:160
#14 0x00007f28730b377b in tbb::internal::market::process (this=0x7f286f5d3580, j=...) at ../../src/tbb/market.cpp:693
#15 0x00007f28730afa26 in tbb::internal::rml::private_worker::run (this=0x7f286f3f5080) at ../../src/tbb/private_server.cpp:270
#16 0x00007f28730afc59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#17 0x00007f2871d70e25 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f2871a9e34d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f284745d700 (LWP 10107)):
#0  0x00007f2871a651ad in nanosleep () from /lib64/libc.so.6
#1  0x00007f2871a65044 in sleep () from /lib64/libc.so.6
#2  0x00007f286d860ade in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f2871a987f9 in syscall () from /lib64/libc.so.6
#5  0x00007f28730afc32 in tbb::internal::futex_wait (comparand=2, futex=0x7f286f3f51ac) at ../../include/tbb/machine/linux_common.h:60
#6  tbb::internal::binary_semaphore::P (this=0x7f286f3f51ac) at ../../src/tbb/semaphore.h:206
#7  rml::internal::thread_monitor::commit_wait (c=<synthetic pointer>..., this=0x7f286f3f51a0) at ../../src/rml/include/../server/thread_monitor.h:259
#8  tbb::internal::rml::private_worker::run (this=0x7f286f3f5180) at ../../src/tbb/private_server.cpp:277
#9  0x00007f28730afc59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007f2871d70e25 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f2871a9e34d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f2847e5e700 (LWP 10106)):
#0  0x00007f2871a651ad in nanosleep () from /lib64/libc.so.6
#1  0x00007f2871a65044 in sleep () from /lib64/libc.so.6
#2  0x00007f286d860ade in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f2871a82e47 in sched_yield () from /lib64/libc.so.6
#5  0x00007f28730ba17f in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7f286f51be00, completion_ref_count=@0x7f286f51ff28: 2, isolation=0) at ../../src/tbb/custom_scheduler.h:299
#6  0x00007f28730bbf5a in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f286f51be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:620
#7  0x00007f28730b4c44 in tbb::internal::arena::process (this=0x7f286f5d2d00, s=...) at ../../src/tbb/arena.cpp:160
#8  0x00007f28730b377b in tbb::internal::market::process (this=0x7f286f5d3580, j=...) at ../../src/tbb/market.cpp:693
#9  0x00007f28730afa26 in tbb::internal::rml::private_worker::run (this=0x7f286f3f5100) at ../../src/tbb/private_server.cpp:270
#10 0x00007f28730afc59 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#11 0x00007f2871d70e25 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f2871a9e34d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f284b4e1700 (LWP 10105)):
#0  0x00007f2871d74945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f28723542dc in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/davidlt/b/BUILD/slc7_amd64_gcc630/external/gcc/6.3.0/gcc-tags_gcc_6_3_0_release-243837/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f284bacb4cb in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&amp)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/biglib/slc7_amd64_gcc630/pluginSimulation.so
#4  0x00007f2872359c3f in std::execute_native_thread_routine (__p=0x7f2856348920) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x00007f2871d70e25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f2871a9e34d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f285fb36700 (LWP 10082)):
#0  0x00007f2871d78189 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f286d860ce7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#2  0x00007f286d861945 in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  0x00007f2872359c3f in std::execute_native_thread_routine (__p=0x7f286f104a80) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007f2871d70e25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f2871a9e34d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f2870168480 (LWP 9980)):
#0  0x00007f2871a651ad in nanosleep () from /lib64/libc.so.6
#1  0x00007f2871a65044 in sleep () from /lib64/libc.so.6
#2  0x00007f286d860ade in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f2874930701 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007f2874930fdf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007f28749358de in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007f287493ce93 in _dl_runtime_resolve_fxsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007f284baf0bd4 in OscarMTProducer::endRun(edm::Run const&amp, edm::EventSetup const&amp) () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/biglib/slc7_amd64_gcc630/pluginSimulation.so
#9  0x00007f2874886e1b in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetup const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#10 0x00007f28747a7cf0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetup const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#11 0x00007f2874786097 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#12 0x00007f28747862dc in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#13 0x00007f28747864b4 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#14 0x00007f287478657d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#15 0x00007f28730bc45c in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f286f5c2600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:509
#16 0x00007f287481223f in edm::EventProcessor::endRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#17 0x00007f2874812333 in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#18 0x00007f287481246b in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#19 0x00007f2874817e6d in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02532/slc7_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-07-13-2300/lib/slc7_amd64_gcc630/libFWCoreFramework.so
#20 0x000000000040e712 in main::{lambda()#1}::operator()() const ()
#21 0x000000000040d12a in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: OscarMTProducer:g4SimHits

A fatal system signal has occurred: abort signal

[Dr15Jones]

Another one of these has occurred https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc700/CMSSW_10_5_ROOT614_X_2019-01-18-2300/pyRelValMatrixLogs/run/7.4_Cosmics_UP18+Cosmics_UP18+DIGICOSPEAK_UP18+RECOCOSPEAK_UP18+ALCACOS_UP18+HARVESTCOS_UP18/step1_Cosmics_UP18+Cosmics_UP18+DIGICOSPEAK_UP18+RECOCOSPEAK_UP18+ALCACOS_UP18+HARVESTCOS_UP18.log#/

cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/9f26d6b67962647923721fcca8c0c3ab/opt/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_5_ROOT614_X_2019-01-16-2300/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Sat Jan 19 04:30:47 CET 2019
Thread 6 (Thread 0x7feffcfff700 (LWP 26657)):
#0  0x00007ff02639ef0d in poll () from /lib64/libc.so.6
#1  0x00007ff021be6547 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007ff021be6bdc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007ff021be7c47 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007ff0262e1277 in raise () from /lib64/libc.so.6
#6  0x00007ff0262e2968 in abort () from /lib64/libc.so.6
#7  0x00007ff0262da096 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007ff0262da142 in __assert_fail () from /lib64/libc.so.6
#9  0x00007ff0298a8466 in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_5_ROOT614_X_2019-01-16-2300/lib/slc7_amd64_gcc700/libFWCoreConcurrency.so
#10 0x00007ff0297303a9 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007ff0296ce29d in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007ff0279c0176 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7ff023cdbe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#13 0x00007ff0279b9a60 in tbb::internal::arena::process (this=0x7ff023dd2d00, s=...) at ../../src/tbb/arena.cpp:160
#14 0x00007ff0279b83e3 in tbb::internal::market::process (this=0x7ff023dd3580, j=...) at ../../src/tbb/market.cpp:710
#15 0x00007ff0279b45e3 in tbb::internal::rml::private_worker::run (this=0x7ff023bf5080) at ../../src/tbb/private_server.cpp:270
#16 0x00007ff0279b4849 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#17 0x00007ff02667fe25 in start_thread () from /lib64/libpthread.so.0
#18 0x00007ff0263a9bad in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7ff000233700 (LWP 26652)):
#0  0x00007ff02637056d in nanosleep () from /lib64/libc.so.6
#1  0x00007ff026370404 in sleep () from /lib64/libc.so.6
#2  0x00007ff021be5e11 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ff0263a3ec9 in syscall () from /lib64/libc.so.6
#5  0x00007ff0279b4805 in tbb::internal::futex_wait (comparand=2, futex=0x7ff023bf51ac) at ../../include/tbb/machine/linux_common.h:85
#6  tbb::internal::binary_semaphore::P (this=0x7ff023bf51ac) at ../../src/tbb/semaphore.h:209
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7ff023bf51a0) at ../../src/tbb/../rml/server/thread_monitor.h:258
#8  tbb::internal::rml::private_worker::run (this=0x7ff023bf5180) at ../../src/tbb/private_server.cpp:277
#9  0x00007ff0279b4849 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007ff02667fe25 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ff0263a9bad in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7ff000c34700 (LWP 26651)):
#0  0x00007ff02637056d in nanosleep () from /lib64/libc.so.6
#1  0x00007ff026370404 in sleep () from /lib64/libc.so.6
#2  0x00007ff021be5e11 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ff0263a3ec9 in syscall () from /lib64/libc.so.6
#5  0x00007ff0279b4805 in tbb::internal::futex_wait (comparand=2, futex=0x7ff023bf512c) at ../../include/tbb/machine/linux_common.h:85
#6  tbb::internal::binary_semaphore::P (this=0x7ff023bf512c) at ../../src/tbb/semaphore.h:209
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7ff023bf5120) at ../../src/tbb/../rml/server/thread_monitor.h:258
#8  tbb::internal::rml::private_worker::run (this=0x7ff023bf5100) at ../../src/tbb/private_server.cpp:277
#9  0x00007ff0279b4849 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007ff02667fe25 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ff0263a9bad in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7ff003b00700 (LWP 26646)):
#0  0x00007ff026683995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ff026c63cec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_1_0_pre2-slc7_amd64_gcc700/build/CMSSW_10_1_0_pre2-build/BUILD/slc7_amd64_gcc700/external/gcc/7.0.0-omkpbe2/gcc-branches_gcc-7-branch-257125/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007ff0040fe82b in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&amp)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/biglib/slc7_amd64_gcc700/pluginSimulation.so
#4  0x00007ff026c698ff in std::execute_native_thread_routine (__p=0x7ff00ad9ad80) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x00007ff02667fe25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ff0263a9bad in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7ff012dbc700 (LWP 26631)):
#0  0x00007ff026687279 in waitpid () from /lib64/libpthread.so.0
#1  0x00007ff021be6017 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007ff021be6afa in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007ff026c698ff in std::execute_native_thread_routine (__p=0x7ff0136f98c0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007ff02667fe25 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ff0263a9bad in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7ff0249e4480 (LWP 26583)):
#0  0x00007ff02637056d in nanosleep () from /lib64/libc.so.6
#1  0x00007ff026370404 in sleep () from /lib64/libc.so.6
#2  0x00007ff021be5e11 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ff0263a3ec9 in syscall () from /lib64/libc.so.6
#5  0x00007ff0279b43ed in tbb::internal::futex_wakeup_one (futex=0x7ff023bf50ac) at ../../include/tbb/machine/linux_common.h:94
#6  tbb::internal::binary_semaphore::V (this=0x7ff023bf50ac) at ../../src/tbb/semaphore.h:218
#7  rml::internal::thread_monitor::notify (this=0x7ff023bf50a0) at ../../src/tbb/../rml/server/thread_monitor.h:243
#8  tbb::internal::rml::private_worker::wake_or_launch (this=0x7ff023bf5080) at ../../src/tbb/private_server.cpp:316
#9  tbb::internal::rml::private_server::wake_some (this=<optimized out>, additional_slack=<optimized out>) at ../../src/tbb/private_server.cpp:399
#10 0x00007ff0279ba6ae in tbb::internal::arena::enqueue_task (this=0x7ff023dd2d00, t=..., prio=<optimized out>, random=...) at ../../src/tbb/arena.cpp:620
#11 0x00007ff0296c5caf in void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007ff02975948c in void edm::endStreamsTransitionAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>, edm::RunPrincipal, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> > >(edm::WaitingTaskHolder, edm::Schedule&amp, unsigned int, edm::RunPrincipal&amp, edm::IOVSyncValue const&amp, edm::EventSetup const&amp, edm::ServiceToken const&amp, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> >&amp, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#13 0x00007ff02974f02e in edm::EventProcessor::endRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007ff02974f13b in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&amp, unsigned int, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007ff02974f2bf in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x0000000000413786 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#17 0x00007ff0297532e3 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02559/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_10_5_ROOT614_X_2019-01-18-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#18 0x000000000040fe8a in main::{lambda()#1}::operator()() const ()
#19 0x000000000040e162 in main ()

[Dr15Jones]

Seen again https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_10_6_DEVEL_X_2019-03-18-2300/pyRelValMatrixLogs/run/106.0_bJpsiX+bJpsiX+DIGI+RECO/step1_bJpsiX+bJpsiX+DIGI+RECO.log#/

Thread 6 (Thread 0x7f0c2cdff700 (LWP 19707)):
#0  0x00007f0c57768e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c57768cc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c52b6fd90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  __cxxabiv1::__class_type_info::__dyncast_result::__dyncast_result (details_=<optimized out>, this=<optimized out>) at ../../../../libstdc++-v3/libsupc++/dyncast.cc:50
#5  __cxxabiv1::__dynamic_cast (src_ptr=0x7f0c2f6ebc90, src_type=0x7f0c5a675000 <typeinfo for edm::Principal>, dst_type=0x7f0c5a66b6b8 <typeinfo for edm::RunPrincipal>, src2dst=0) at ../../../../libstdc++-v3/libsupc++/dyncast.cc:57
#6  0x00007f0c5a55b80d in edm::Run::runPrincipal() const () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#7  0x00007f0c5a55b849 in edm::Run::index() const () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#8  0x00007f0c5a5db1c5 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#9  0x00007f0c5a528810 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#10 0x00007f0c5a50ac65 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007f0c5a50ae89 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007f0c5a50b024 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f0c5a50b0fd in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007f0c58ddc7a1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0c5503be00, parent=..., child=<optimized out>) at ../../src/tbb/scheduler.cpp:1020
#15 0x00007f0c58dd6498 in tbb::internal::arena::process (this=0x7f0c5513ad00, s=...) at ../../src/tbb/arena.cpp:160
#16 0x00007f0c58dd4f23 in tbb::internal::market::process (this=0x7f0c5513b580, j=...) at ../../src/tbb/market.cpp:710
#17 0x00007f0c58dd150c in tbb::internal::rml::private_worker::run (this=0x7f0c54f5e080) at ../../src/tbb/private_server.cpp:270
#18 0x00007f0c58dd1719 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#19 0x00007f0c57a78dd5 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f0c577a1ead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f0c2e02d700 (LWP 19706)):
#0  0x00007f0c5779720d in poll () from /lib64/libc.so.6
#1  0x00007f0c52b703af in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f0c52b70aec in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f0c52b719c9 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f0c576da207 in raise () from /lib64/libc.so.6
#6  0x00007f0c576db8f8 in abort () from /lib64/libc.so.6
#7  0x00007f0c576d3026 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007f0c576d30d2 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f0c5a6bbf6e in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreConcurrency.so
#10 0x00007f0c5a57761b in edm::Worker::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007f0c5a575f94 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007f0c5a50b937 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetupImpl const&, edm::ServiceToken const&, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f0c58ddc7a1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0c55063e00, parent=..., child=<optimized out>) at ../../src/tbb/scheduler.cpp:1020
#14 0x00007f0c58dd6498 in tbb::internal::arena::process (this=0x7f0c5513ad00, s=...) at ../../src/tbb/arena.cpp:160
#15 0x00007f0c58dd4f23 in tbb::internal::market::process (this=0x7f0c5513b580, j=...) at ../../src/tbb/market.cpp:710
#16 0x00007f0c58dd150c in tbb::internal::rml::private_worker::run (this=0x7f0c54f5e180) at ../../src/tbb/private_server.cpp:270
#17 0x00007f0c58dd1719 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#18 0x00007f0c57a78dd5 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f0c577a1ead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f0c2ea2e700 (LWP 19705)):
#0  0x00007f0c57768e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c57768cc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c52b6fd90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c5779c1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f0c58dd16e5 in tbb::internal::futex_wait (comparand=2, futex=0x7f0c54f5e12c) at ../../include/tbb/machine/linux_common.h:85
#6  tbb::internal::binary_semaphore::P (this=0x7f0c54f5e12c) at ../../src/tbb/semaphore.h:209
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f0c54f5e120) at ../../src/tbb/../rml/server/thread_monitor.h:258
#8  tbb::internal::rml::private_worker::run (this=0x7f0c54f5e100) at ../../src/tbb/private_server.cpp:277
#9  0x00007f0c58dd1719 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007f0c57a78dd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f0c577a1ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f0c31d20700 (LWP 19698)):
#0  0x00007f0c57a7c965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0c580605ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc820/external/gcc/8.2.0-lafael/gcc-8.2.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f0c324001a3 in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const [clone .isra.82] () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/biglib/slc7_amd64_gcc820/pluginSimulation.so
#4  0x00007f0c58065ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#5  0x00007f0c57a78dd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f0c577a1ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f0c45103700 (LWP 19670)):
#0  0x00007f0c57a80179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f0c52b6ff47 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f0c52b70a0a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f0c58065ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007f0c57a78dd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f0c577a1ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f0c55e65480 (LWP 19555)):
#0  0x00007f0c57768e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c57768cc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c52b6fd90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c5a4fd5f7 in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::~FunctorTask() () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#5  0x00007f0c58ddc51a in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0c5512a600, parent=..., child=<optimized out>) at ../../include/tbb/task.h:940
#6  0x00007f0c5a59252a in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#7  0x00007f0c5a5925eb in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&, unsigned int, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#8  0x00007f0c5a59275f in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#9  0x00007f0c5a595e36 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02568/slc7_amd64_gcc820/cms/cmssw/CMSSW_10_6_DEVEL_X_2019-03-18-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#10 0x0000000000410c7b in main::{lambda()#1}::operator()() const ()
#11 0x000000000040f2e2 in main ()

Current Modules:

Module: none (crashed)
Module: OscarMTProducer:g4SimHits
Module: none
Module: none

[Dr15Jones]

This time, two asserts failed

Begin processing the 500th record. Run 1, Event 500, LumiSection 1 on stream 2 at 31-Mar-2019 09:41:16.636 CEST
%MSG-w MemoryCheck:  CosMuoGenProducer:generator 31-Mar-2019 09:41:16 CEST  Run: 1 Event: 498
MemoryCheck: module CosMuoGenProducer:generator VSIZE 1848.68 0 RSS 673.453 0.0390625
%MSG
cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/4d494545ae8f4df30c7264f50697bc4a/opt/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.
cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/4d494545ae8f4df30c7264f50697bc4a/opt/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Sun Mar 31 09:41:16 CEST 2019
Thread 6 (Thread 0x7fe64bbff700 (LWP 10781)):
#0  0x00007fe69e5fce2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fe69e5fccc4 in sleep () from /lib64/libc.so.6
#2  0x00007fe6993f6670 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe6a30746f5 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fe6a3074fcf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007fe6a3079d1e in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007fe6a308191a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007fe657164613 in OscarMTProducer::endRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/pluginSimG4CoreApplicationPlugins.so
#9  0x00007fe6a2dc1947 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00007fe6a2cf063d in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007fe6a2cb5d9e in edm::workerhelper::CallImpl<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >::call(edm::Worker*, edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::StreamContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007fe6a2cb5c27 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#13 0x00007fe6a2cb59e9 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007fe6a2cb581d in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007fe6a2cb68aa in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00007fe69fc4c176 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe69bf1be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#17 0x00007fe69fc45a60 in tbb::internal::arena::process (this=0x7fe69bfcad00, s=...) at ../../src/tbb/arena.cpp:160
#18 0x00007fe69fc443e3 in tbb::internal::market::process (this=0x7fe69bfcb580, j=...) at ../../src/tbb/market.cpp:710
#19 0x00007fe69fc405e3 in tbb::internal::rml::private_worker::run (this=0x7fe69bdee080) at ../../src/tbb/private_server.cpp:270
#20 0x00007fe69fc40849 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#21 0x00007fe69e90cdd5 in start_thread () from /lib64/libpthread.so.0
#22 0x00007fe69e635ead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fe64ca67700 (LWP 10780)):
#0  0x00007fe69e5fce2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fe69e5fccc4 in sleep () from /lib64/libc.so.6
#2  0x00007fe6993f6670 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe69e63036a in mmap64 () from /lib64/libc.so.6
#5  0x00007fe69e566fe4 in __assert_fail_base () from /lib64/libc.so.6
#6  0x00007fe69e5670d2 in __assert_fail () from /lib64/libc.so.6
#7  0x00007fe6a11aae5b in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreConcurrency.so
#8  0x00007fe6a2d32db7 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#9  0x00007fe6a2cb47cc in void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetupImpl const&, edm::ServiceToken const&, bool)::{lambda()#1}::operator()() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00007fe6a2cb473a in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetupImpl const&, edm::ServiceToken const&, bool)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007fe69fc4c176 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe69bf0be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#12 0x00007fe69fc45a60 in tbb::internal::arena::process (this=0x7fe69bfcad00, s=...) at ../../src/tbb/arena.cpp:160
#13 0x00007fe69fc443e3 in tbb::internal::market::process (this=0x7fe69bfcb580, j=...) at ../../src/tbb/market.cpp:710
#14 0x00007fe69fc405e3 in tbb::internal::rml::private_worker::run (this=0x7fe69bdee180) at ../../src/tbb/private_server.cpp:270
#15 0x00007fe69fc40849 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#16 0x00007fe69e90cdd5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007fe69e635ead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fe64d468700 (LWP 10779)):
#0  0x00007fe69e5fce2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fe69e5fccc4 in sleep () from /lib64/libc.so.6
#2  0x00007fe6993f6670 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe6a30746c7 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fe6a3074fcf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007fe6a3079d1e in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007fe6a308191a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007fe657164613 in OscarMTProducer::endRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/pluginSimG4CoreApplicationPlugins.so
#9  0x00007fe6a2dc1947 in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00007fe6a2cf063d in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007fe6a2cb5d9e in edm::workerhelper::CallImpl<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >::call(edm::Worker*, edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*, edm::StreamContext const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007fe6a2cb5c27 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#13 0x00007fe6a2cb59e9 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007fe6a2cb581d in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007fe6a2cb68aa in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00007fe69fc4c176 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe69bf23e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#17 0x00007fe69fc45a60 in tbb::internal::arena::process (this=0x7fe69bfcad00, s=...) at ../../src/tbb/arena.cpp:160
#18 0x00007fe69fc443e3 in tbb::internal::market::process (this=0x7fe69bfcb580, j=...) at ../../src/tbb/market.cpp:710
#19 0x00007fe69fc405e3 in tbb::internal::rml::private_worker::run (this=0x7fe69bdee100) at ../../src/tbb/private_server.cpp:270
#20 0x00007fe69fc40849 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#21 0x00007fe69e90cdd5 in start_thread () from /lib64/libpthread.so.0
#22 0x00007fe69e635ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fe652d06700 (LWP 10778)):
#0  0x00007fe69e910965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe69eef0cec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_1_0_pre2-slc7_amd64_gcc700/build/CMSSW_10_1_0_pre2-build/BUILD/slc7_amd64_gcc700/external/gcc/7.0.0-omkpbe2/gcc-branches_gcc-7-branch-257125/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007fe656f2986b in std::thread::_State_impl<std::thread::_Invoker<std::tuple<OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::$_0> > >::_M_run() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libSimG4CoreApplication.so
#4  0x00007fe69eef68ff in std::execute_native_thread_routine (__p=0x7fe689300540) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x00007fe69e90cdd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fe69e635ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fe689127700 (LWP 10279)):
#0  0x00007fe69e914179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007fe6993f725d in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007fe69eef68ff in std::execute_native_thread_routine (__p=0x7fe689398aa0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#3  0x00007fe69e90cdd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fe69e635ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fe69ccfd480 (LWP 8582)):
#0  0x00007fe69e62b20d in poll () from /lib64/libc.so.6
#1  0x00007fe6993f754c in (anonymous namespace)::full_read(int, char*, unsigned long, int) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007fe6993f700a in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007fe6993f69ef in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fe69e56e207 in raise () from /lib64/libc.so.6
#6  0x00007fe69e56f8f8 in abort () from /lib64/libc.so.6
#7  0x00007fe69e567026 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007fe69e5670d2 in __assert_fail () from /lib64/libc.so.6
#9  0x00007fe6a11aae5b in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreConcurrency.so
#10 0x00007fe6a2d32db7 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007fe6a2cb47cc in void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetupImpl const&, edm::ServiceToken const&, bool)::{lambda()#1}::operator()() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007fe6a2cb473a in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetupImpl const&, edm::ServiceToken const&, bool)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#13 0x00007fe69fc4c176 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe69bfba600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#14 0x00007fe6a2d6518e in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007fe6a2d64e07 in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&, unsigned int, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00007fe6a2d6a47d in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#17 0x00007fe6a2d635ac in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_CLANG_X_2019-03-31-0000/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#18 0x000000000040cb89 in main::$_0::operator()() const ()
#19 0x000000000040a458 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: OscarMTProducer:g4SimHits
Module: OscarMTProducer:g4SimHits

A fatal system signal has occurred: abort signal

[wddgit]

I have a theory about this. UnscheduledCallProducer::runNowAsync and Path::runAllModulesAsync both contain loops where they are starting asynchronous work for multiple Workers. But I do not see anything that forces the loop to finish before the successor WaitingTask is started. So in one iteration of the loop a Worker gets going, finishes, and tells the WaitingTask to go and if it has finished before the next Worker says "Hey wait for me also", then execution can proceed to endRun before all the Workers have finished streamEndLumi. What do you think? Should I put in a fix for this?

[Dr15Jones]

UnscheduledCallProducer::runNowAsync is called ultimately by StreamSchedule::processOneStreamAsync on line 451. That is within another task which has the doneTask passed to UnscheduledCallProducer::runNowAsync being held by a WaitingTaskHolder in the outer task.

[Dr15Jones]

Path::runAllModulesAsync is also called by that same outer task only line 444 and 448.

[wddgit]

You are right. I saw the lambda in processOneStreamAsync capturing doneTask but missed that it is also capturing the WaitingTaskHolder(doneTask). Must be something else causing the problem. Sorry for the noise.

[Dr15Jones]

Another case has happened https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc700/CMSSW_10_6_ROOT6_X_2019-04-16-2300/pyRelValMatrixLogs/run/106.0_bJpsiX+bJpsiX+DIGI+RECO/step1_bJpsiX+bJpsiX+DIGI+RECO.log#/

 *-------  PYTHIA Error and Warning Messages Statistics  ----------------------------------------------------------* 
 |                                                                                                                 | 
 |  times   message                                                                                                | 
 |                                                                                                                 | 
 |      0   no errors or warnings to report                                                                        | 
 |                                                                                                                 | 
 *-------  End PYTHIA Error and Warning Messages Statistics  ------------------------------------------------------* 
cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c331ad1bab9c5bab5ea7a134f53bbdd0/opt/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Wed Apr 17 02:29:52 CEST 2019
Thread 6 (Thread 0x7f622c5ff700 (LWP 18429)):
#0  0x00007f6255399e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f6255399cc4 in sleep () from /lib64/libc.so.6
#2  0x00007f62507c6061 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f62588fc6f5 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007f62588fcfcf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007f6258901d1e in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007f625890991a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007f6230967994 in OscarMTProducer::endRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/biglib/slc7_amd64_gcc700/pluginSimulation.so
#9  0x00007f62588522cb in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00007f6258773dd0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f6258742345 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f6258742563 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#13 0x00007f6258742794 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007f625874286d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007f62569ec931 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f6252acbe00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#16 0x00007f62569e63d8 in tbb::internal::arena::process (this=0x7f6252bcad00, s=...) at ../../src/tbb/arena.cpp:160
#17 0x00007f62569e4d53 in tbb::internal::market::process (this=0x7f6252bcb580, j=...) at ../../src/tbb/market.cpp:710
#18 0x00007f62569e1011 in tbb::internal::rml::private_worker::run (this=0x7f62529ee080) at ../../src/tbb/private_server.cpp:270
#19 0x00007f62569e1259 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#20 0x00007f62556a9dd5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f62553d2ead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f622d664700 (LWP 18428)):
#0  0x00007f6255399e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f6255399cc4 in sleep () from /lib64/libc.so.6
#2  0x00007f62507c6061 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f625890da71 in mempcpy () from /lib64/ld-linux-x86-64.so.2
#5  0x00007f6258904f84 in tls_get_addr_tail () from /lib64/ld-linux-x86-64.so.2
#6  0x00007f625890a928 in __tls_get_addr () from /lib64/ld-linux-x86-64.so.2
#7  0x00007f624f53eed3 in CLHEP::(anonymous namespace)::theDefaults () at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc700/external/clhep/2.4.0.0-pafccj/clhep-2.4.0.0/Random/src/Random.cc:157
#8  0x00007f624f53f439 in CLHEP::HepRandom::getTheEngine () at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc700/external/clhep/2.4.0.0-pafccj/clhep-2.4.0.0/Random/src/Random.cc:268
#9  0x00007f623096797e in OscarMTProducer::endRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/biglib/slc7_amd64_gcc700/pluginSimulation.so
#10 0x00007f62588522cb in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f6258773dd0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f6258742345 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#13 0x00007f6258742563 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007f6258742794 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007f625874286d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00007f62569ec931 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f6252b1be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#17 0x00007f62569e63d8 in tbb::internal::arena::process (this=0x7f6252bcad00, s=...) at ../../src/tbb/arena.cpp:160
#18 0x00007f62569e4d53 in tbb::internal::market::process (this=0x7f6252bcb580, j=...) at ../../src/tbb/market.cpp:710
#19 0x00007f62569e1011 in tbb::internal::rml::private_worker::run (this=0x7f62529ee180) at ../../src/tbb/private_server.cpp:270
#20 0x00007f62569e1259 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#21 0x00007f62556a9dd5 in start_thread () from /lib64/libpthread.so.0
#22 0x00007f62553d2ead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f622e065700 (LWP 18427)):
#0  0x00007f6255399e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f6255399cc4 in sleep () from /lib64/libc.so.6
#2  0x00007f62507c6061 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f62588fc6cb in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007f62588fcfcf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007f6258901d1e in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007f625890991a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007f6230967994 in OscarMTProducer::endRun(edm::Run const&, edm::EventSetup const&) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/biglib/slc7_amd64_gcc700/pluginSimulation.so
#9  0x00007f62588522cb in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00007f6258773dd0 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f6258742345 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f6258742563 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#13 0x00007f6258742794 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007f625874286d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007f62569ec931 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f6252b23e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#16 0x00007f62569e63d8 in tbb::internal::arena::process (this=0x7f6252bcad00, s=...) at ../../src/tbb/arena.cpp:160
#17 0x00007f62569e4d53 in tbb::internal::market::process (this=0x7f6252bcb580, j=...) at ../../src/tbb/market.cpp:710
#18 0x00007f62569e1011 in tbb::internal::rml::private_worker::run (this=0x7f62529ee100) at ../../src/tbb/private_server.cpp:270
#19 0x00007f62569e1259 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#20 0x00007f62556a9dd5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f62553d2ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f62304bb700 (LWP 18398)):
#0  0x00007f62556ad965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f6255c8e0ac in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/gcc-branches_gcc-7-branch-268351/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f623098fa6b in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/biglib/slc7_amd64_gcc700/pluginSimulation.so
#4  0x00007f6255c93cbf in std::execute_native_thread_routine (__p=0x7f623bc44280) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x00007f62556a9dd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f62553d2ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f624276e700 (LWP 18259)):
#0  0x00007f62556b1179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f62507c6267 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f62507c6d4a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f6255c93cbf in std::execute_native_thread_routine (__p=0x7f6242c01d70) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007f62556a9dd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f62553d2ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f6253a0d480 (LWP 18153)):
#0  0x00007f62553c820d in poll () from /lib64/libc.so.6
#1  0x00007f62507c6797 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007f62507c6e2c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007f62507c7e98 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f625530b207 in raise () from /lib64/libc.so.6
#6  0x00007f625530c8f8 in abort () from /lib64/libc.so.6
#7  0x00007f6255304026 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007f62553040d2 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f62589362f6 in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreConcurrency.so
#10 0x00007f62587b8859 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007f62587431ed in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetupImpl const&, edm::ServiceToken const&, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007f62569ec931 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f6252bba600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#13 0x00007f62587ec59a in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007f62587ec68b in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&, unsigned int, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007f62587ec80f in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00000000004137b6 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#17 0x00007f62587f07c8 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#18 0x000000000040fe78 in main::{lambda()#1}::operator()() const ()
#19 0x000000000040e142 in main ()

Current Modules:

Module: none (crashed)
Module: OscarMTProducer:g4SimHits
Module: OscarMTProducer:g4SimHits
Module: OscarMTProducer:g4SimHits

[Dr15Jones]

And yet another one: https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc700/CMSSW_10_6_ROOT6_X_2019-04-16-2300/pyRelValMatrixLogs/run/122.0_UpsMM+UpsMM+DIGI+RECO/step1_UpsMM+UpsMM+DIGI+RECO.log#/

 *-------  PYTHIA Error and Warning Messages Statistics  ----------------------------------------------------------* 
 |                                                                                                                 | 
 |  times   message                                                                                                | 
 |                                                                                                                 | 
 |      0   no errors or warnings to report                                                                        | 
 |                                                                                                                 | 
 *-------  End PYTHIA Error and Warning Messages Statistics  ------------------------------------------------------* 
cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/c331ad1bab9c5bab5ea7a134f53bbdd0/opt/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/src/FWCore/Concurrency/src/WaitingTaskList.cc:58: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Wed Apr 17 02:32:53 CEST 2019
Thread 6 (Thread 0x7ff7bf1ff700 (LWP 19295)):
#0  0x00007ff7e7d6e20d in poll () from /lib64/libc.so.6
#1  0x00007ff7e316c797 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007ff7e316ce2c in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007ff7e316de98 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007ff7e7cb1207 in raise () from /lib64/libc.so.6
#6  0x00007ff7e7cb28f8 in abort () from /lib64/libc.so.6
#7  0x00007ff7e7caa026 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007ff7e7caa0d2 in __assert_fail () from /lib64/libc.so.6
#9  0x00007ff7eb2dc2f6 in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreConcurrency.so
#10 0x00007ff7eb15e859 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007ff7eb0e91ed in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetupImpl const&, edm::ServiceToken const&, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007ff7e9392931 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7ff7e5503e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:521
#13 0x00007ff7e938c3d8 in tbb::internal::arena::process (this=0x7ff7e55cad00, s=...) at ../../src/tbb/arena.cpp:160
#14 0x00007ff7e938ad53 in tbb::internal::market::process (this=0x7ff7e55cb580, j=...) at ../../src/tbb/market.cpp:710
#15 0x00007ff7e9387011 in tbb::internal::rml::private_worker::run (this=0x7ff7e53ee080) at ../../src/tbb/private_server.cpp:270
#16 0x00007ff7e9387259 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#17 0x00007ff7e804fdd5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007ff7e7d78ead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7ff7c0608700 (LWP 19292)):
#0  0x00007ff7e7d3fe2d in nanosleep () from /lib64/libc.so.6
#1  0x00007ff7e7d3fcc4 in sleep () from /lib64/libc.so.6
#2  0x00007ff7e316c061 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ff7e7d731c9 in syscall () from /lib64/libc.so.6
#5  0x00007ff7e9387215 in tbb::internal::futex_wait (comparand=2, futex=0x7ff7e53ee1ac) at ../../include/tbb/machine/linux_common.h:85
#6  tbb::internal::binary_semaphore::P (this=0x7ff7e53ee1ac) at ../../src/tbb/semaphore.h:209
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7ff7e53ee1a0) at ../../src/tbb/../rml/server/thread_monitor.h:258
#8  tbb::internal::rml::private_worker::run (this=0x7ff7e53ee180) at ../../src/tbb/private_server.cpp:277
#9  0x00007ff7e9387259 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007ff7e804fdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ff7e7d78ead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7ff7c1009700 (LWP 19291)):
#0  0x00007ff7e7d3fe2d in nanosleep () from /lib64/libc.so.6
#1  0x00007ff7e7d3fcc4 in sleep () from /lib64/libc.so.6
#2  0x00007ff7e316c061 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ff7e7d731c9 in syscall () from /lib64/libc.so.6
#5  0x00007ff7e9387215 in tbb::internal::futex_wait (comparand=2, futex=0x7ff7e53ee12c) at ../../include/tbb/machine/linux_common.h:85
#6  tbb::internal::binary_semaphore::P (this=0x7ff7e53ee12c) at ../../src/tbb/semaphore.h:209
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7ff7e53ee120) at ../../src/tbb/../rml/server/thread_monitor.h:258
#8  tbb::internal::rml::private_worker::run (this=0x7ff7e53ee100) at ../../src/tbb/private_server.cpp:277
#9  0x00007ff7e9387259 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007ff7e804fdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ff7e7d78ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7ff7c345f700 (LWP 19280)):
#0  0x00007ff7e8053965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007ff7e86340ac in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/jenkins/workspace/build-any-ib/w/BUILD/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/gcc-branches_gcc-7-branch-268351/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007ff7c3933a6b in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/biglib/slc7_amd64_gcc700/pluginSimulation.so
#4  0x00007ff7e8639cbf in std::execute_native_thread_routine (__p=0x7ff7d34141e0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x00007ff7e804fdd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007ff7e7d78ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7ff7d5111700 (LWP 19218)):
#0  0x00007ff7e8057179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007ff7e316c267 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007ff7e316cd4a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007ff7e8639cbf in std::execute_native_thread_routine (__p=0x7ff7d62ced70) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007ff7e804fdd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007ff7e7d78ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7ff7e63b3480 (LWP 19116)):
#0  0x00007ff7e7d3fe2d in nanosleep () from /lib64/libc.so.6
#1  0x00007ff7e7d3fcc4 in sleep () from /lib64/libc.so.6
#2  0x00007ff7e316c061 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007ff7e7d5dd47 in sched_yield () from /lib64/libc.so.6
#5  0x00007ff7e9391715 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7ff7e55ba600, completion_ref_count=<optimized out>, isolation=0) at ../../src/tbb/custom_scheduler.h:289
#6  0x00007ff7e9392508 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7ff7e55ba600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:633
#7  0x00007ff7eb19259a in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#8  0x00007ff7eb19268b in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&, unsigned int, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#9  0x00007ff7eb19280f in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00000000004137b6 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#11 0x00007ff7eb1967c8 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02572/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_ROOT6_X_2019-04-15-2300/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x000000000040fe78 in main::{lambda()#1}::operator()() const ()
#13 0x000000000040e142 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: none

A fatal system signal has occurred: abort signal

[Dr15Jones]

And yet another one https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_11_0_X_2019-05-27-2300/pyRelValMatrixLogs/run/7.2_Cosmics_UP18+Cosmics_UP18+DIGICOS_UP18+RECOCOS_UP18+ALCACOS_UP18+HARVESTCOS_UP18/step1_Cosmics_UP18+Cosmics_UP18+DIGICOS_UP18+RECOCOS_UP18+ALCACOS_UP18+HARVESTCOS_UP18.log#/

Begin processing the 500th record. Run 1, Event 500, LumiSection 1 on stream 0 at 28-May-2019 04:21:24.687 CEST
%MSG-w MemoryCheck:  OscarMTProducer:g4SimHits 28-May-2019 04:21:26 CEST  Run: 1 Event: 352
MemoryCheck: module OscarMTProducer:g4SimHits VSIZE 1161.15 0 RSS 575.078 7.26172
%MSG
%MSG-w MemoryCheck:  PoolOutputModule:RAWSIMoutput 28-May-2019 04:21:26 CEST  Run: 1 Event: 352
MemoryCheck: module PoolOutputModule:RAWSIMoutput VSIZE 1161.15 0 RSS 575.277 0.199219
%MSG
cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/9df1e9526a1f91b89d27536a3dc17168/opt/cmssw/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/src/FWCore/Concurrency/src/WaitingTaskList.cc:55: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Tue May 28 04:21:26 CEST 2019
Thread 6 (Thread 0x7f157c7ff700 (LWP 35173)):
#0  0x00007f15a31ac20d in poll () from /lib64/libc.so.6
#1  0x00007f159e97d3af in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f159e97daec in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f159e97e9c9 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f15a30ef207 in raise () from /lib64/libc.so.6
#6  0x00007f15a30f08f8 in abort () from /lib64/libc.so.6
#7  0x00007f15a30e8026 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007f15a30e80d2 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f15a6204f6e in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/libFWCoreConcurrency.so
#10 0x00007f15a603b04b in edm::Worker::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007f15a60399a4 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007f15a5fca707 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetupImpl const&, edm::ServiceToken const&, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f15a47f17a1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f15a0ac3e00, parent=..., child=<optimized out>) at ../../src/tbb/scheduler.cpp:1020
#14 0x00007f15a47eb498 in tbb::internal::arena::process (this=0x7f15a0b7ad00, s=...) at ../../src/tbb/arena.cpp:160
#15 0x00007f15a47e9f23 in tbb::internal::market::process (this=0x7f15a0b7b580, j=...) at ../../src/tbb/market.cpp:710
#16 0x00007f15a47e650c in tbb::internal::rml::private_worker::run (this=0x7f15a099e080) at ../../src/tbb/private_server.cpp:270
#17 0x00007f15a47e6719 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#18 0x00007f15a348ddd5 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f15a31b6ead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f157dbfe700 (LWP 35172)):
#0  0x00007f15a317de2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f15a317dcc4 in sleep () from /lib64/libc.so.6
#2  0x00007f159e97cd90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f15a31b11c9 in syscall () from /lib64/libc.so.6
#5  0x00007f15a47e66e5 in tbb::internal::futex_wait (comparand=2, futex=0x7f15a099e1ac) at ../../include/tbb/machine/linux_common.h:85
#6  tbb::internal::binary_semaphore::P (this=0x7f15a099e1ac) at ../../src/tbb/semaphore.h:209
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f15a099e1a0) at ../../src/tbb/../rml/server/thread_monitor.h:258
#8  tbb::internal::rml::private_worker::run (this=0x7f15a099e180) at ../../src/tbb/private_server.cpp:277
#9  0x00007f15a47e6719 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007f15a348ddd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f15a31b6ead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f157e5ff700 (LWP 35171)):
#0  0x00007f15a317de2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f15a317dcc4 in sleep () from /lib64/libc.so.6
#2  0x00007f159e97cd90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f15a31b11c9 in syscall () from /lib64/libc.so.6
#5  0x00007f15a47e66e5 in tbb::internal::futex_wait (comparand=2, futex=0x7f15a099e12c) at ../../include/tbb/machine/linux_common.h:85
#6  tbb::internal::binary_semaphore::P (this=0x7f15a099e12c) at ../../src/tbb/semaphore.h:209
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f15a099e120) at ../../src/tbb/../rml/server/thread_monitor.h:258
#8  tbb::internal::rml::private_worker::run (this=0x7f15a099e100) at ../../src/tbb/private_server.cpp:277
#9  0x00007f15a47e6719 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:223
#10 0x00007f15a348ddd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f15a31b6ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f1581c92700 (LWP 35155)):
#0  0x00007f15a3491965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f15a3a755ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc820/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/gcc-8.2.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f158220cce3 in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&)::{lambda()#1}::operator()() const [clone .isra.78] () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/biglib/slc7_amd64_gcc820/pluginSimulation.so
#4  0x00007f15a3a7accf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#5  0x00007f15a348ddd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f15a31b6ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f15906e2700 (LWP 35106)):
#0  0x00007f15a3495179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f159e97cf47 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f159e97da0a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f15a3a7accf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007f15a348ddd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f15a31b6ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f15a1878480 (LWP 34921)):
#0  0x00007f15a317de2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f15a317dcc4 in sleep () from /lib64/libc.so.6
#2  0x00007f159e97cd90 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f15a31b11c9 in syscall () from /lib64/libc.so.6
#5  0x00007f15a47e6351 in tbb::internal::futex_wakeup_one (futex=0x7f15a099e0ac) at ../../include/tbb/machine/linux_common.h:94
#6  tbb::internal::binary_semaphore::V (this=0x7f15a099e0ac) at ../../src/tbb/semaphore.h:218
#7  tbb::internal::binary_semaphore::V (this=0x7f15a099e0ac) at ../../src/tbb/semaphore.h:215
#8  rml::internal::thread_monitor::notify (this=0x7f15a099e0a0) at ../../src/tbb/../rml/server/thread_monitor.h:243
#9  tbb::internal::rml::private_worker::wake_or_launch (this=0x7f15a099e080) at ../../src/tbb/private_server.cpp:316
#10 tbb::internal::rml::private_server::wake_some(int) () at ../../src/tbb/private_server.cpp:399
#11 0x00007f15a47ee5e2 in tbb::internal::generic_scheduler::local_spawn(tbb::task*, tbb::task*&) () at ../../src/tbb/scheduler.cpp:681
#12 0x00007f15a5fc1a09 in void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&, edm::EventSetupImpl const&, edm::ServiceToken const&, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f15a6070627 in void edm::endStreamsTransitionAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>, edm::RunPrincipal, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> > >(edm::WaitingTaskHolder, edm::Schedule&, unsigned int, edm::RunPrincipal&, edm::IOVSyncValue const&, edm::EventSetupImpl const&, edm::ServiceToken const&, std::vector<edm::SubProcess, std::allocator<edm::SubProcess> >&, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007f15a606974e in edm::EventProcessor::endRun(edm::Hash<2> const&, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007f15a606982b in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&, unsigned int, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#16 0x00007f15a606999f in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#17 0x00007f15a606d810 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02578/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#18 0x0000000000410c7b in main::{lambda()#1}::operator()() const ()
#19 0x000000000040f2e2 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: none
Module: none

A fatal system signal has occurred: abort signal
cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/9df1e9526a1f91b89d27536a3dc17168/opt/cmssw/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-05-27-2300/src/FWCore/Concurrency/src/WaitingTaskList.cc:55: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

This one is interesting since the main thread is in the process of doing a spawn, all except one of the TBB worker threads is idle and the one non-idle TBB worker has the assert.

[makortel]

Here is another one https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc700/CMSSW_11_0_X_2019-08-14-2300/pyRelValMatrixLogs/run/7.1_CosmicsSPLoose+CosmicsSPLoose+DIGICOS+RECOCOS+ALCACOS+HARVESTCOS/step1_CosmicsSPLoose+CosmicsSPLoose+DIGICOS+RECOCOS+ALCACOS+HARVESTCOS.log#/2681-2681

Begin processing the 1999th record. Run 1, Event 1999, LumiSection 1 on stream 1 at 15-Aug-2019 04:53:57.612 CEST
%MSG-w MemoryCheck:  GenParticleProducer:genParticles 15-Aug-2019 04:53:57 CEST  Run: 1 Event: 1996
MemoryCheck: module GenParticleProducer:genParticles VSIZE 1162.52 0 RSS 553.734 0.140625
%MSG
%MSG-w MemoryCheck:  InputGenJetsParticleSelector:genParticlesForJets  15-Aug-2019 04:53:57 CEST Run: 1 Event: 1996
MemoryCheck: module InputGenJetsParticleSelector:genParticlesForJets VSIZE 1162.52 0 RSS 553.84 0.105469
%MSG
%MSG-w MemoryCheck:  FastjetJetProducer:ak4GenJets 15-Aug-2019 04:53:57 CEST  Run: 1 Event: 1996
MemoryCheck: module FastjetJetProducer:ak4GenJets VSIZE 1162.52 0 RSS 554.027 0.1875
%MSG
%MSG-w MemoryCheck:  GenMETProducer:genMetCalo 15-Aug-2019 04:53:57 CEST  Run: 1 Event: 1996
MemoryCheck: module GenMETProducer:genMetCalo VSIZE 1162.52 0 RSS 554.047 0.0195312
%MSG
%MSG-w MemoryCheck:  CosMuoGenProducer:generator 15-Aug-2019 04:53:57 CEST  Run: 1 Event: 1997
MemoryCheck: module CosMuoGenProducer:generator VSIZE 1162.52 0 RSS 554.176 0.128906
%MSG
Begin processing the 2000th record. Run 1, Event 2000, LumiSection 1 on stream 2 at 15-Aug-2019 04:53:57.662 CEST
%MSG-w MemoryCheck:  CosMuoGenProducer:generator 15-Aug-2019 04:53:57 CEST  Run: 1 Event: 1998
MemoryCheck: module CosMuoGenProducer:generator VSIZE 1162.52 0 RSS 554.793 0.617188
%MSG
%MSG-w MemoryCheck:  CosMuoGenProducer:generator 15-Aug-2019 04:53:57 CEST  Run: 1 Event: 1999
MemoryCheck: module CosMuoGenProducer:generator VSIZE 1227.52 65 RSS 565.965 11.1719
%MSG
%MSG-w MemoryCheck:  CosmicGenFilterHelix:cosmicInPixelLoose  15-Aug-2019 04:53:57 CEST Run: 1 Event: 2000
MemoryCheck: module CosmicGenFilterHelix:cosmicInPixelLoose VSIZE 1227.52 0 RSS 565.969 0.00390625
%MSG
%MSG-w MemoryCheck:  OscarMTProducer:g4SimHits 15-Aug-2019 04:54:00 CEST  Run: 1 Event: 1996
MemoryCheck: module OscarMTProducer:g4SimHits VSIZE 1252.52 25 RSS 597.953 31.9844
%MSG
%MSG-w MemoryCheck:  PoolOutputModule:RAWSIMoutput 15-Aug-2019 04:54:00 CEST  Run: 1 Event: 1996
MemoryCheck: module PoolOutputModule:RAWSIMoutput VSIZE 1252.52 0 RSS 598.414 0.460938
%MSG
cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/83355b38ac88742c2a26b57f19c31d36/opt/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/src/FWCore/Concurrency/src/WaitingTaskList.cc:55: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Thu Aug 15 04:54:00 CEST 2019
Thread 9 (Thread 0x7fc2f57ff700 (LWP 237371)):
#0  0x00007fc33a9d5e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fc33a9d5cc4 in sleep () from /lib64/libc.so.6
#2  0x00007fc33585d2e1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fc33d6146f5 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fc33d614fcf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007fc33d619d1e in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007fc33d62191a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007fc318b0ba79 in RunAction::~RunAction() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_11_0_X_2019-08-14-2300/biglib/slc7_amd64_gcc700/pluginSimulation.so
#9  0x00007fc318b0c72c in RunManagerMTWorker::terminateRun() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_11_0_X_2019-08-14-2300/biglib/slc7_amd64_gcc700/pluginSimulation.so
#10 0x00007fc318adff74 in OscarMTProducer::endRun(edm::Run const&amp, edm::EventSetup const&amp) () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_11_0_X_2019-08-14-2300/biglib/slc7_amd64_gcc700/pluginSimulation.so
#11 0x00007fc33d567ffb in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007fc33d484970 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#13 0x00007fc33d452b15 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x00007fc33d452d33 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#15 0x00007fc33d452f64 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#16 0x00007fc33d45303d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#17 0x00007fc33bc91a71 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fc338043e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:517
#18 0x00007fc33bc8b508 in tbb::internal::arena::process (this=0x7fc3381cad00, s=...) at ../../src/tbb/arena.cpp:156
#19 0x00007fc33bc89e63 in tbb::internal::market::process (this=0x7fc3381cb580, j=...) at ../../src/tbb/market.cpp:702
#20 0x00007fc33bc86121 in tbb::internal::rml::private_worker::run (this=0x7fc337eecf00) at ../../src/tbb/private_server.cpp:266
#21 0x00007fc33bc86369 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#22 0x00007fc33ace5dd5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fc33aa0eead in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fc2f6afe700 (LWP 237366)):
#0  0x00007fc33a9d5e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fc33a9d5cc4 in sleep () from /lib64/libc.so.6
#2  0x00007fc33585d2e1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fc33aa091c9 in syscall () from /lib64/libc.so.6
#5  0x00007fc33bc86325 in tbb::internal::futex_wait (comparand=2, futex=0x7fc337eed02c) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7fc337eed02c) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7fc337eed020) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7fc337eed000) at ../../src/tbb/private_server.cpp:273
#9  0x00007fc33bc86369 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007fc33ace5dd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fc33aa0eead in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fc2f74ff700 (LWP 237365)):
#0  0x00007fc33a9d5e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fc33a9d5cc4 in sleep () from /lib64/libc.so.6
#2  0x00007fc33585d2e1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fc33acea7a0 in pthread_getspecific () from /lib64/libpthread.so.0
#5  0x00007fc33bc8f24a in tbb::internal::basic_tls<unsigned long>::get (this=<optimized out>) at ../../src/tbb/tls.h:43
#6  tbb::internal::governor::local_scheduler () at ../../src/tbb/governor.h:123
#7  tbb::internal::generic_scheduler::spawn (this=<optimized out>, first=..., next=@0x7fc3381bdf38: 0x7fc338091b40) at ../../src/tbb/scheduler.cpp:703
#8  0x00007fc33d684ff2 in edm::WaitingTaskList::announce() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreConcurrency.so
#9  0x00007fc33d452f89 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00007fc33d45303d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007fc33bc91a71 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fc33806be00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:517
#12 0x00007fc33bc8b508 in tbb::internal::arena::process (this=0x7fc3381cad00, s=...) at ../../src/tbb/arena.cpp:156
#13 0x00007fc33bc89e63 in tbb::internal::market::process (this=0x7fc3381cb580, j=...) at ../../src/tbb/market.cpp:702
#14 0x00007fc33bc86121 in tbb::internal::rml::private_worker::run (this=0x7fc337eecf80) at ../../src/tbb/private_server.cpp:266
#15 0x00007fc33bc86369 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#16 0x00007fc33ace5dd5 in start_thread () from /lib64/libpthread.so.0
#17 0x00007fc33aa0eead in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fc312dff700 (LWP 237211)):
#0  0x00007fc33a9d5e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fc33a9d5cc4 in sleep () from /lib64/libc.so.6
#2  0x00007fc33585d2e1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fc33aa091c9 in syscall () from /lib64/libc.so.6
#5  0x00007fc33bc86325 in tbb::internal::futex_wait (comparand=2, futex=0x7fc337eed0ac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7fc337eed0ac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7fc337eed0a0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7fc337eed080) at ../../src/tbb/private_server.cpp:273
#9  0x00007fc33bc86369 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007fc33ace5dd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fc33aa0eead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fc313c07700 (LWP 237210)):
#0  0x00007fc33a9d5e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fc33a9d5cc4 in sleep () from /lib64/libc.so.6
#2  0x00007fc33585d2e1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fc33aa091c9 in syscall () from /lib64/libc.so.6
#5  0x00007fc33bc86325 in tbb::internal::futex_wait (comparand=2, futex=0x7fc337eed1ac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7fc337eed1ac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7fc337eed1a0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7fc337eed180) at ../../src/tbb/private_server.cpp:273
#9  0x00007fc33bc86369 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007fc33ace5dd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fc33aa0eead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fc314608700 (LWP 237209)):
#0  0x00007fc33aa0420d in poll () from /lib64/libc.so.6
#1  0x00007fc33585da17 in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007fc33585e0ac in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007fc33585f118 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fc33a947207 in raise () from /lib64/libc.so.6
#6  0x00007fc33a9488f8 in abort () from /lib64/libc.so.6
#7  0x00007fc33a940026 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007fc33a9400d2 in __assert_fail () from /lib64/libc.so.6
#9  0x00007fc33d684e16 in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreConcurrency.so
#10 0x00007fc33d4ca699 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007fc33d4539bd in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00007fc33bc91a71 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fc338113e00, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:517
#13 0x00007fc33bc8b508 in tbb::internal::arena::process (this=0x7fc3381cad00, s=...) at ../../src/tbb/arena.cpp:156
#14 0x00007fc33bc89e63 in tbb::internal::market::process (this=0x7fc3381cb580, j=...) at ../../src/tbb/market.cpp:702
#15 0x00007fc33bc86121 in tbb::internal::rml::private_worker::run (this=0x7fc337eed100) at ../../src/tbb/private_server.cpp:266
#16 0x00007fc33bc86369 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#17 0x00007fc33ace5dd5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fc33aa0eead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fc3174b2700 (LWP 237208)):
#0  0x00007fc33ace9965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fc33b2ca0ac in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc700/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/gcc-branches_gcc-7-branch-268351/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007fc318b0a45b in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&amp)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw-patch/CMSSW_11_0_X_2019-08-14-2300/biglib/slc7_amd64_gcc700/pluginSimulation.so
#4  0x00007fc33b2cfcbf in std::execute_native_thread_routine (__p=0x7fc31ec87c20) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#5  0x00007fc33ace5dd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fc33aa0eead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fc326687700 (LWP 237179)):
#0  0x00007fc33aced179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007fc33585d4e7 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#2  0x00007fc33585dfca in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  0x00007fc33b2cfcbf in std::execute_native_thread_routine (__p=0x7fc326de7cf0) at ../../../../../libstdc++-v3/src/c++11/thread.cc:83
#4  0x00007fc33ace5dd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fc33aa0eead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fc338ebf4c0 (LWP 237153)):
#0  0x00007fc33a9d5e2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fc33a9d5cc4 in sleep () from /lib64/libc.so.6
#2  0x00007fc33585d2e1 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fc33bc90715 in __TBB_machine_pause (delay=4) at ../../include/tbb/machine/gcc_ia32_common.h:45
#5  tbb::internal::atomic_backoff::bounded_pause (this=<synthetic pointer>) at ../../include/tbb/tbb_machine.h:373
#6  tbb::internal::prolonged_pause () at ../../src/tbb/scheduler_common.h:305
#7  tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7fc3381ba600, completion_ref_count=<optimized out>, isolation=0) at ../../src/tbb/custom_scheduler.h:277
#8  0x00007fc33bc91648 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fc3381ba600, parent=..., child=<optimized out>) at ../../src/tbb/custom_scheduler.h:629
#9  0x00007fc33d4ff42a in edm::EventProcessor::endRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#10 0x00007fc33d4ff51b in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&amp, unsigned int, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#11 0x00007fc33d4ff69f in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#12 0x00000000004137c6 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#13 0x00007fc33d503708 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02589/slc7_amd64_gcc700/cms/cmssw/CMSSW_11_0_X_2019-08-13-1100/lib/slc7_amd64_gcc700/libFWCoreFramework.so
#14 0x000000000040fe88 in main::{lambda()#1}::operator()() const ()
#15 0x000000000040e152 in main ()

Current Modules:

Module: none (crashed)
Module: none
Module: OscarMTProducer:g4SimHits
Module: none
Module: none

A fatal system signal has occurred: abort signal

[Dr15Jones]

This happened again https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_11_0_ROOT618_X_2019-09-10-0900/pyRelValMatrixLogs/run/7.21_Cosmics_UP17+Cosmics_UP17+DIGICOS_UP17+RECOCOS_UP17+ALCACOS_UP17+HARVESTCOS_UP17/step1_Cosmics_UP17+Cosmics_UP17+DIGICOS_UP17+RECOCOS_UP17+ALCACOS_UP17+HARVESTCOS_UP17.log#/

cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/d6b2ffc5b898403a08663e2c36c347a7/opt/cmssw/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/src/FWCore/Concurrency/src/WaitingTaskList.cc:55: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Tue Sep 10 17:16:25 CEST 2019
Thread 6 (Thread 0x7fe86edff700 (LWP 47867)):
#0  0x00007fe896d2b38d in poll () from /lib64/libc.so.6
#1  0x00007fe891ed73bf in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007fe891ed7afc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007fe891ed89d9 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fe896c6e2c7 in raise () from /lib64/libc.so.6
#6  0x00007fe896c6f9b8 in abort () from /lib64/libc.so.6
#7  0x00007fe896c670e6 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007fe896c67192 in __assert_fail () from /lib64/libc.so.6
#9  0x00007fe899a17eee in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreConcurrency.so
#10 0x00007fe8998735bb in edm::Worker::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007fe899871f14 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007fe8997fe497 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007fe897fba8a1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe89447be00, parent=..., child=<optimized out>) at ../../src/tbb/scheduler.cpp:1016
#14 0x00007fe897fb4588 in tbb::internal::arena::process (this=0x7fe8945d2d00, s=...) at ../../src/tbb/arena.cpp:156
#15 0x00007fe897fb2ff3 in tbb::internal::market::process (this=0x7fe8945d3580, j=...) at ../../src/tbb/market.cpp:702
#16 0x00007fe897faf5dc in tbb::internal::rml::private_worker::run (this=0x7fe89503c080) at ../../src/tbb/private_server.cpp:266
#17 0x00007fe897faf7e9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#18 0x00007fe89700cdd5 in start_thread () from /lib64/libpthread.so.0
#19 0x00007fe896d3602d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fe86fd17700 (LWP 47866)):
#0  0x00007fe896cfcfad in nanosleep () from /lib64/libc.so.6
#1  0x00007fe896cfce44 in sleep () from /lib64/libc.so.6
#2  0x00007fe891ed6da0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe8999b96a2 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fe8999b9fcf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007fe8999bed1e in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007fe8999c691a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007fe87494df24 in OscarMTProducer::endRun(edm::Run const&amp, edm::EventSetup const&amp) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/biglib/slc7_amd64_gcc820/pluginSimulation.so
#9  0x00007fe8999007db in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#10 0x00007fe899832510 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007fe8997fd7c5 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007fe8997fd9e9 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007fe8997fdb84 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007fe8997fdc5d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007fe897fba8a1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe894473e00, parent=..., child=<optimized out>) at ../../src/tbb/scheduler.cpp:1016
#16 0x00007fe897fb4588 in tbb::internal::arena::process (this=0x7fe8945d2d00, s=...) at ../../src/tbb/arena.cpp:156
#17 0x00007fe897fb2ff3 in tbb::internal::market::process (this=0x7fe8945d3580, j=...) at ../../src/tbb/market.cpp:702
#18 0x00007fe897faf5dc in tbb::internal::rml::private_worker::run (this=0x7fe89503c180) at ../../src/tbb/private_server.cpp:266
#19 0x00007fe897faf7e9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#20 0x00007fe89700cdd5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fe896d3602d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fe870718700 (LWP 47865)):
#0  0x00007fe896cfcfad in nanosleep () from /lib64/libc.so.6
#1  0x00007fe896cfce44 in sleep () from /lib64/libc.so.6
#2  0x00007fe891ed6da0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe8999b96cb in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fe8999b9fcf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007fe8999bed1e in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007fe8999c691a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007fe87494df24 in OscarMTProducer::endRun(edm::Run const&amp, edm::EventSetup const&amp) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/biglib/slc7_amd64_gcc820/pluginSimulation.so
#9  0x00007fe8999007db in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#10 0x00007fe899832510 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007fe8997fd7c5 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007fe8997fd9e9 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007fe8997fdb84 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007fe8997fdc5d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007fe897fba8a1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe89448be00, parent=..., child=<optimized out>) at ../../src/tbb/scheduler.cpp:1016
#16 0x00007fe897fb4588 in tbb::internal::arena::process (this=0x7fe8945d2d00, s=...) at ../../src/tbb/arena.cpp:156
#17 0x00007fe897fb2ff3 in tbb::internal::market::process (this=0x7fe8945d3580, j=...) at ../../src/tbb/market.cpp:702
#18 0x00007fe897faf5dc in tbb::internal::rml::private_worker::run (this=0x7fe89503c100) at ../../src/tbb/private_server.cpp:266
#19 0x00007fe897faf7e9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#20 0x00007fe89700cdd5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fe896d3602d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fe873207700 (LWP 47813)):
#0  0x00007fe897010965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fe8975f45ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc820/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/gcc-8.2.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007fe8749690e3 in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&amp)::{lambda()#1}::operator()() const [clone .isra.88] () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/biglib/slc7_amd64_gcc820/pluginSimulation.so
#4  0x00007fe8975f9ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#5  0x00007fe89700cdd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fe896d3602d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fe883d38700 (LWP 47749)):
#0  0x00007fe897014179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007fe891ed6f57 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007fe891ed7a1a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007fe8975f9ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007fe89700cdd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fe896d3602d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fe8951ea4c0 (LWP 47603)):
#0  0x00007fe896cfcfad in nanosleep () from /lib64/libc.so.6
#1  0x00007fe896cfce44 in sleep () from /lib64/libc.so.6
#2  0x00007fe891ed6da0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fe8999b96f1 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007fe8999b9fcf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007fe8999bed1e in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007fe8999c691a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007fe87494df24 in OscarMTProducer::endRun(edm::Run const&amp, edm::EventSetup const&amp) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/biglib/slc7_amd64_gcc820/pluginSimulation.so
#9  0x00007fe8999007db in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#10 0x00007fe899832510 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007fe8997fd7c5 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007fe8997fd9e9 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007fe8997fdb84 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007fe8997fdc5d in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007fe897fba8a1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fe8945c2600, parent=..., child=<optimized out>) at ../../src/tbb/scheduler.cpp:1016
#16 0x00007fe8998a57f8 in edm::EventProcessor::endRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#17 0x00007fe8998a58bb in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&amp, unsigned int, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#18 0x00007fe8998a5a2f in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#19 0x00007fe8998a99b0 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02593/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_ROOT618_X_2019-09-10-0900/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#20 0x0000000000410c7b in main::{lambda()#1}::operator()() const ()
#21 0x000000000040f2e2 in main ()

Current Modules:

Module: none (crashed)
Module: OscarMTProducer:g4SimHits
Module: OscarMTProducer:g4SimHits
Module: OscarMTProducer:g4SimHits

A fatal system signal has occurred: abort signal

[wddgit]

A commit in PR #27948 might help with this, maybe... Commit title "Be more careful about resetting lumi status". It might help because the lumi status objects were being held in functors held up in queues behind other functors that need to run and potentially causing endRun to be started very late. I am not sure about this and did not trace this through in detail enough yet to connect it to these assert failures...

[wddgit]

So I am guessing the cause is that in RunsInFileProcessor::processRun there is no guarantee that something else does not have a copy of the RunResources shared_ptr and that could potentially delay the endRun past the start of the next run. There is a data race where the correct code path almost always wins. This line is where the shared_ptr gets overwritten, causing endRun to execute on the old run.

currentRun_ = std::make_shared<RunResources>(iEP, runID.first, runID.second);

But if one of the lumi or stream queues still has a functor that is holding onto a lumi status object that contains a shared_ptr to RunResources, then even if that functor ends up doing nothing (because its work was already done by a task from another stream queue or concurrent lumi queue) and it seems like it doesn't matter whether it has run yet or not, it does matter. The PR makes sure all the shared_ptr's to RunResources are reset before getting to this point. It explicitly resets the pointer to the RunResources object and does not wait for the destructor of the lumi status object to run. So the theory would be the next run is starting and has contents in the WaitingTaskList of the Workers when the old run tries to reset them for the previous run... I have not verified this, but at least it seems plausible. We will see if the errors go away.

[Dr15Jones]

This happened again in CMSSW_11_0_X_2019-10-04-2300 for workflow 7.23 step1:

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_11_0_X_2019-10-04-2300/pyRelValMatrixLogs/run/7.23_Cosmics_UP21+Cosmics_UP21+DIGICOS_UP21+RECOCOS_UP21+ALCACOS_UP21+HARVESTCOS_UP21/step1_Cosmics_UP21+Cosmics_UP21+DIGICOS_UP21+RECOCOS_UP21+ALCACOS_UP21+HARVESTCOS_UP21.log#/

cmsRun: /build/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/231ff565af1e00974b78b43856de0a34/opt/cmssw/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/src/FWCore/Concurrency/src/WaitingTaskList.cc:55: void edm::WaitingTaskList::reset(): Assertion `m_head == nullptr' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Sat Oct  5 01:19:14 CEST 2019
Thread 6 (Thread 0x7f53fe3ff700 (LWP 32449)):
#0  0x00007f542669efad in nanosleep () from /lib64/libc.so.6
#1  0x00007f542669ee44 in sleep () from /lib64/libc.so.6
#2  0x00007f5421326da0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f54266bcec7 in sched_yield () from /lib64/libc.so.6
#5  0x00007f5427955007 in tbb::internal::market::process (this=0x7f5423dcb580, j=...) at ../../src/tbb/market.cpp:710
#6  0x00007f54279515dc in tbb::internal::rml::private_worker::run (this=0x7f5423aed080) at ../../src/tbb/private_server.cpp:266
#7  0x00007f54279517e9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#8  0x00007f54269aedd5 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f54266d802d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f53ff89c700 (LWP 32448)):
#0  0x00007f54266cd38d in poll () from /lib64/libc.so.6
#1  0x00007f54213273bf in full_read.constprop () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f5421327afc in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f54213289d9 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f54266102c7 in raise () from /lib64/libc.so.6
#6  0x00007f54266119b8 in abort () from /lib64/libc.so.6
#7  0x00007f54266090e6 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007f5426609192 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f542935aeee in edm::WaitingTaskList::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreConcurrency.so
#10 0x00007f54291b562b in edm::Worker::reset() () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007f54291b3f84 in edm::WorkerManager::resetAll() () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007f542913f707 in edm::FunctorTask<void edm::StreamSchedule::processOneStreamAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTaskHolder, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, bool)::{lambda()#2}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f542795c8a1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f5423c73e00, parent=..., child=<optimized out>) at ../../src/tbb/scheduler.cpp:1016
#14 0x00007f5427956588 in tbb::internal::arena::process (this=0x7f5423dcad00, s=...) at ../../src/tbb/arena.cpp:156
#15 0x00007f5427954ff3 in tbb::internal::market::process (this=0x7f5423dcb580, j=...) at ../../src/tbb/market.cpp:702
#16 0x00007f54279515dc in tbb::internal::rml::private_worker::run (this=0x7f5423aed180) at ../../src/tbb/private_server.cpp:266
#17 0x00007f54279517e9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#18 0x00007f54269aedd5 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f54266d802d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f540029d700 (LWP 32447)):
#0  0x00007f542669efad in nanosleep () from /lib64/libc.so.6
#1  0x00007f542669ee44 in sleep () from /lib64/libc.so.6
#2  0x00007f5421326da0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f54292fc6f5 in do_lookup_x () from /lib64/ld-linux-x86-64.so.2
#5  0x00007f54292fcfcf in _dl_lookup_symbol_x () from /lib64/ld-linux-x86-64.so.2
#6  0x00007f5429301d1e in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#7  0x00007f542930991a in _dl_runtime_resolve_xsave () from /lib64/ld-linux-x86-64.so.2
#8  0x00007f54037b5664 in OscarMTProducer::endRun(edm::Run const&amp, edm::EventSetup const&amp) () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_0_X_2019-10-04-2300/biglib/slc7_amd64_gcc820/pluginSimulation.so
#9  0x00007f5429242f4b in edm::stream::ProducingModuleAdaptorBase<edm::stream::EDProducerBase>::doStreamEndRun(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#10 0x00007f5429173960 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDoStreamEnd(edm::StreamID, edm::RunPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ModuleCallingContext const*) () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007f542913ea35 in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}) () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007f542913ec59 in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*) () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f542913edf4 in void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}::operator()() const () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007f542913eecd in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007f542795c8a1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f5423c83e00, parent=..., child=<optimized out>) at ../../src/tbb/scheduler.cpp:1016
#16 0x00007f5427956588 in tbb::internal::arena::process (this=0x7f5423dcad00, s=...) at ../../src/tbb/arena.cpp:156
#17 0x00007f5427954ff3 in tbb::internal::market::process (this=0x7f5423dcb580, j=...) at ../../src/tbb/market.cpp:702
#18 0x00007f54279515dc in tbb::internal::rml::private_worker::run (this=0x7f5423aed100) at ../../src/tbb/private_server.cpp:266
#19 0x00007f54279517e9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#20 0x00007f54269aedd5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f54266d802d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f5403193700 (LWP 32440)):
#0  0x00007f54269b2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f5426f965ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc820/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/gcc-8.2.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f54037d0673 in OscarMTMasterThread::OscarMTMasterThread(edm::ParameterSet const&amp)::{lambda()#1}::operator()() const [clone .isra.88] () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_0_X_2019-10-04-2300/biglib/slc7_amd64_gcc820/pluginSimulation.so
#4  0x00007f5426f9bccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#5  0x00007f54269aedd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f54266d802d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f5413372700 (LWP 32418)):
#0  0x00007f54269b6179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f5421326f57 in edm::service::cmssw_stacktrace_fork() () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f5421327a1a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f5426f9bccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007f54269aedd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f54266d802d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f5424b844c0 (LWP 32396)):
#0  0x00007f542669efad in nanosleep () from /lib64/libc.so.6
#1  0x00007f542669ee44 in sleep () from /lib64/libc.so.6
#2  0x00007f5421326da0 in sig_pause_for_stacktrace () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f542913eec0 in edm::FunctorTask<void edm::Worker::doWorkNoPrefetchingAsync<edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::MyPrincipal const&amp, edm::EventSetupImpl const&amp, edm::ServiceToken const&amp, edm::StreamID, edm::ParentContext const&amp, edm::OccurrenceTraits<edm::RunPrincipal, (edm::BranchActionType)2>::Context const*)::{lambda()#1}>::execute() () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#5  0x00007f542795c8a1 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f5423dba600, parent=..., child=<optimized out>) at ../../src/tbb/scheduler.cpp:1016
#6  0x00007f54291eff18 in edm::EventProcessor::endRun(edm::Hash<2> const&amp, unsigned int, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#7  0x00007f54291effeb in edm::EventProcessor::endUnfinishedRun(edm::Hash<2> const&amp, unsigned int, bool, bool, bool) () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#8  0x00007f54291f015f in std::_Sp_counted_ptr_inplace<edm::(anonymous namespace)::RunResources, std::allocator<edm::(anonymous namespace)::RunResources>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#9  0x00007f54291f4100 in edm::EventProcessor::runToCompletion() () from /cvmfs/cms-ib.cern.ch/nweek-02596/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_X_2019-10-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#10 0x0000000000410c7b in main::{lambda()#1}::operator()() const ()
#11 0x000000000040f2e2 in main ()

Current Modules:

Module: none (crashed)
Module: OscarMTProducer:g4SimHits
Module: none
Module: none

A fatal system signal has occurred: abort signal

[makortel]

This doesn't seem to have occurred for a long time, maybe we could close it?

[Dr15Jones]

This doesn't seem to have occurred for a long time, maybe we could close it?

Sure. Do you want to do the 'signature'?

[makortel]

+1

[cmsbuild]

This issue is fully signed and ready to be closed.