Raw data object rarely goes missing in multi-threaded HLT jobs

[smorovic]

During recent output module tests which involve a lot of CPU time spent in output modules (for 2021 Heavy Ion running), I have been encountering spurious errors with 4-thread and 4-stream cmsRun jobs and would like to report it.

The issue appears when a module can not find product inserted by the input source (invalid handle). This causes following error message in the L1 HLT module (found in VirginRaw HLT menu):

%MSG-e L1T:  L1TRawToDigi:hltGtStage2Digis 03-Mar-2020 18:59:42 CET  Run: 1000001680 Event: 6996208
Cannot unpack: no FEDRawDataCollection found
%MSG

(followed by other fatal exception).

It happens here: https://github.com/cms-sw/cmssw/blob/master/EventFilter/L1TRawToDigi/plugins/L1TRawToDigi.cc#L140

It is relatively rare and hard to reproduce. I was running on a scaled-down HLT farm of 12 nodes (32 cores per node, 1 proces assigned per 4 cores). In this case problem appears every 30 minutes. It also only appears when CPU load is near maximum (which is achieved using expensive LZMA compression in the output module or doing other equivalent work with similar CPU time, as I tested). It did not appear with 1-stream/thread setup. While I first saw this with CMSSW_11_0_1, further testing showed also the same problem with CMSSW_10_6_8.

I did some debugging, and it appears code in FedRawDataInputSource which inserts RawDataCollection does run (according to logged message) before this issue appears in the same event, i.e. it seems that the product goes missing in between.

[smorovic]

It seems I didn't enable LZMA compression in the HLT menu in the package, which would load CPU more. It can be done by uncommenting this from near the end of HltConfig.py:

    if C_ALGO_VALUE != C_ALGO_UNDEFINED:
        modified_module.compression_algorithm=cms.untracked.string(C_ALGO_VALUE)

However I was getting the exception very easily even when using default gzip one. I also got one crash with LZMA already.

[smorovic]

I will keep trying. It is indeed strange that first trace had 19 threads, but I don't understand why. Latter cases all had 12 and this is normal (there are a few input source threads for prefetching data + one for monitoring snapshots in addition to configured tbb threads).

[Dr15Jones]

It is indeed strange that first trace had 19 threads, but I don't understand why. Latter cases all had 12 and this is normal (there are a few input source threads for prefetching data + one for monitoring snapshots in addition to configured tbb threads).

Do you call any ROOT routines from a thread you started? In particular, any ROOT routines dealing with I/O? Some of those routines call TBB under the hood. If you call it from a non-TBB thread, TBB will startup more threads to do the work requested.

[Dr15Jones]

If the abort happens quickly, you could try adding

process.add_(cms.Service("Tracer"))

to the job to see what is happening. The down side is the log grows extremely quickly.

[Dr15Jones]

So looking at the second traceback and the results of my own Tracer job I see that hltOutputPhysicsVirginRaw4 and hltOutputPhysicsVirginRaw7 both depend on data products from hltTriggerSummaryAOD which depends on data products from hltGtStage2Digis. Therefore, if things are working correctly, the OutputModules are processing data from a different event.

[smorovic]

Yes, I think that should be the case. Each hltPreRandom* has different offset (0 to 15) and prescale factors are 16. Each output module takes 2 of those paths. So events are distributed in round robin fashion between output modules / streams (it makes it difficult to reduce the size of the menu and keep threads saturated doing compression).

I also tried to run without HLTriggerFinalPath and RatesMonitoring but didn't get any occurrence. I got it when I again switched on HLTriggerFinalPath. I will try to repeat that and bisect which module affects it.

Also, No ROOT routines should be used in either of those additional threads in input source and monitoring.

[Dr15Jones]

Do you have the stacktrace to the recen failure when you didn't have RatesMonitoring?

[Dr15Jones]

To get a better understanding of this particular configuration, I added the StallMonitor service and then plotted the results. Here is what I got

The blue part is during global begin run, the light green is during stream begin run then the entire red section is processing after that. The red means those are times where modules which have had significant stall times are being run. Not being able to see anything but red (normally you want to see dark green) means this entire job is completely dominated by stalling modules. The modules with the largest stalls are

Stalled Module             Tot Stall Time  Stall Times
hltOutputPhysicsVirginRaw3: 2579.80        121.02, 109.05, 106.88, 104.83, 104.61, 104.24, 103.89, 103.77, 102.45, 102.28, 102.15, 102.08, 101.72, 101.66, 101.49, 101.38, 101.13, 101.03, 101.00, 100.96, 100.92, 100.74, 100.37, 100.12, 100.04
hltOutputPhysicsVirginRaw4: 1805.46        203.44, 198.28, 195.87, 176.75, 107.63, 106.90, 104.00, 102.79, 102.74, 102.46, 102.02, 101.99, 100.49, 100.10
hltOutputPhysicsVirginRaw2: 1745.08        113.79, 105.78, 103.96, 103.39, 102.75, 102.64, 102.44, 102.27, 102.10, 101.58, 101.57, 101.52, 100.79, 100.28, 100.10, 100.06, 100.05
hltOutputPhysicsVirginRaw1: 1428.62        203.24, 107.16, 105.29, 104.58, 102.58, 102.00, 101.27, 100.79, 100.64, 100.46, 100.31, 100.22, 100.08
hltOutputPhysicsVirginRaw7: 1376.53        204.49, 204.19, 203.46, 140.28, 119.45, 102.43, 101.42, 100.72, 100.08, 100.01
hltOutputPhysicsVirginRaw8: 825.61         107.19, 103.87, 103.69, 103.38, 102.48, 102.31, 102.11, 100.57
hltOutputPhysicsVirginRaw5: 821.25         111.23, 104.53, 102.19, 101.20, 100.87, 100.74, 100.43, 100.06
hltOutputPhysicsVirginRaw6: 305.21         102.69, 101.40, 101.13

So the round robin appears to be insufficient to allow this job to utilize the threads effectively.

[smorovic]

The module which triggers the problem is process.hltFEDSelector:

process.hltFEDSelector = cms.EDProducer( "EvFFEDSelector",
    inputTag = cms.InputTag( "rawDataCollector" ),
    fedList = cms.vuint32( 1023, 1024 )
)

I disabled RatesMonitoring and HLTriggerFinalPath, and added hltFEDSelector to BeginSequence as below.

process.HLTBeginSequenceRandom = cms.Sequence( process.hltRandomEventsFilter + process.hltGtStage2Digis + process.hltFEDSelector)

It resulted in this abort and stacktrace (I think nothing unusual apart from 19 threads with this one)

Mon Mar 23 19:08:06 CET 2020
Thread 18 (Thread 0x7f0bc97fe700 (LWP 316662)):
#0  0x00007f0c62f8ee2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c62f8ecc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c5f114de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c62fc21c9 in syscall () from /lib64/libc.so.6
#5  0x00007f0c642467a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f0c60878c2c) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f0c60878c2c) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f0c60878c20) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f0c60878c00) at ../../src/tbb/private_server.cpp:273
#9  0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f0bca1ff700 (LWP 316661)):
#0  0x00007f0c62f8ee2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c62f8ecc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c5f114de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c64250352 in __TBB_machine_pause (delay=8) at ../../include/tbb/machine/gcc_ia32_common.h:46
#5  tbb::internal::atomic_backoff::bounded_pause (this=<synthetic pointer>) at ../../include/tbb/tbb_machine.h:373
#6  tbb::internal::prolonged_pause () at ../../src/tbb/scheduler_common.h:306
#7  tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7f0c3d2bbe00, completion_ref_count=<optimized out>, isolation=0) at ../../src/tbb/custom_scheduler.h:305
#8  0x00007f0c6424b2be in tbb::internal::arena::process (this=0x7f0c60b37480, s=...) at ../../include/tbb/task.h:992
#9  0x00007f0c64249ed3 in tbb::internal::market::process (this=0x7f0c60b57580, j=...) at ../../src/tbb/market.cpp:667
#10 0x00007f0c642465cc in tbb::internal::rml::private_worker::run (this=0x7f0c60878b80) at ../../src/tbb/private_server.cpp:266
#11 0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#12 0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f0bccd7c700 (LWP 316198)):
#0  0x00007f0c62fc21c9 in syscall () from /lib64/libc.so.6
#1  0x00007f0c642467a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f0c60878d2c) at ../../include/tbb/machine/linux_common.h:81
#2  tbb::internal::binary_semaphore::P (this=0x7f0c60878d2c) at ../../src/tbb/semaphore.h:205
#3  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f0c60878d20) at ../../src/tbb/../rml/server/thread_monitor.h:255
#4  tbb::internal::rml::private_worker::run (this=0x7f0c60878d00) at ../../src/tbb/private_server.cpp:273
#5  0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#6  0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f0bcd77d700 (LWP 316197)):
#0  0x00007f0c62f8ee2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c62f8ecc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c5f114de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c62fc21c9 in syscall () from /lib64/libc.so.6
#5  0x00007f0c642467a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f0c60878e2c) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f0c60878e2c) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f0c60878e20) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f0c60878e00) at ../../src/tbb/private_server.cpp:273
#9  0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f0bce17e700 (LWP 316196)):
#0  0x00007f0c62fc21c9 in syscall () from /lib64/libc.so.6
#1  0x00007f0c642467a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f0c60878dac) at ../../include/tbb/machine/linux_common.h:81
#2  tbb::internal::binary_semaphore::P (this=0x7f0c60878dac) at ../../src/tbb/semaphore.h:205
#3  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f0c60878da0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#4  tbb::internal::rml::private_worker::run (this=0x7f0c60878d80) at ../../src/tbb/private_server.cpp:273
#5  0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#6  0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f0bceb7f700 (LWP 316195)):
#0  0x00007f0c62fc21c9 in syscall () from /lib64/libc.so.6
#1  0x00007f0c642467a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f0c60878cac) at ../../include/tbb/machine/linux_common.h:81
#2  tbb::internal::binary_semaphore::P (this=0x7f0c60878cac) at ../../src/tbb/semaphore.h:205
#3  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f0c60878ca0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#4  tbb::internal::rml::private_worker::run (this=0x7f0c60878c80) at ../../src/tbb/private_server.cpp:273
#5  0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#6  0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f0bd25ff700 (LWP 316182)):
#0  0x00007f0c632a3d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0c5e99e9aa in FedRawDataInputSource::readSupervisor() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#2  0x00007f0c6388bccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#3  0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f0c379fe700 (LWP 316175)):
#0  0x00007f0c62f8ee2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c62f8ecc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c5f114de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c62fc21c9 in syscall () from /lib64/libc.so.6
#5  0x00007f0c642467a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f0c60878eac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f0c60878eac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f0c60878ea0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f0c60878e80) at ../../src/tbb/private_server.cpp:273
#9  0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f0c387ff700 (LWP 316169)):
#0  0x00007f0c62f8ee2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c62f8ecc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c5f114de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c62fc21c9 in syscall () from /lib64/libc.so.6
#5  0x00007f0c642467a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f0c60878fac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f0c60878fac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f0c60878fa0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f0c60878f80) at ../../src/tbb/private_server.cpp:273
#9  0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f0c39ffe700 (LWP 316165)):
#0  0x00007f0c62fbd20d in poll () from /lib64/libc.so.6
#1  0x00007f0c5f1153ff in full_read.constprop () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f0c5f115b3c in edm::service::InitRootHandlers::stacktraceFromThread() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f0c5f116a19 in sig_dostack_then_abort () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f0c62f00207 in raise () from /lib64/libc.so.6
#6  0x00007f0c62f018f8 in abort () from /lib64/libc.so.6
#7  0x00007f0c62ef9026 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007f0c62ef90d2 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f0c3f4630c7 in l1t::L1TRawToDigi::produce(edm::Event&, edm::EventSetup const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginEventFilterL1TRawToDigiAuto.so
#10 0x00007f0c65ba0956 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007f0c65ac1b83 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007f0c65a69c8a in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f0c65a69e5e in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007f0c65a6b65b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007f0c65a6c674 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#16 0x00007f0c6425127d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f0c60a27e00, context_guard=..., t=t@entry=0x7f0c60a6dd40, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#17 0x00007f0c64251575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0c60a27e00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#18 0x00007f0c6424b35f in tbb::internal::arena::process (this=0x7f0c60b37480, s=...) at ../../src/tbb/arena.cpp:152
#19 0x00007f0c64249ed3 in tbb::internal::market::process (this=0x7f0c60b57580, j=...) at ../../src/tbb/market.cpp:667
#20 0x00007f0c642465cc in tbb::internal::rml::private_worker::run (this=0x7f0c60878f00) at ../../src/tbb/private_server.cpp:266
#21 0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#22 0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f0c395fd700 (LWP 316163)):
#0  0x00007f0c62f8ee2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c62f8ecc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c5f114de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000000000415406 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) ()
#5  0x00007f0c657ad253 in edm::ProductProvenanceRetriever::branchIDToProvenance(edm::BranchID const&) const [clone .localalias.120] () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libDataFormatsProvenance.so
#6  0x00007f0c5e8ea86e in edm::StreamSerializer::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Hash<1> const&, edm::StreamerCompressionAlgo, int, unsigned int) const () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#7  0x00007f0c5e8f6c13 in edm::StreamerOutputModuleCommon::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Handle<edm::TriggerResults> const&, edm::Hash<1> const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#8  0x00007f0c5e9a865b in evf::EvFOutputModule::write(edm::EventForOutput const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#9  0x00007f0c65b9dc9f in edm::one::OutputModuleBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#10 0x00007f0c65ac1543 in edm::WorkerT<edm::one::OutputModuleBase>::implDo(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007f0c65a69c8a in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007f0c65a69e5e in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f0c65a6b65b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007f0c65a6ccc5 in void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007f0c65a6cd81 in edm::SerialTaskQueue::QueuedTask<void edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#16 0x00007f0c6425127d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f0c60a2fe00, context_guard=..., t=0x7f0c60a00340, t@entry=0x7f0c60a31d40, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#17 0x00007f0c64251575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0c60a2fe00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#18 0x00007f0c6424b35f in tbb::internal::arena::process (this=0x7f0c60b37480, s=...) at ../../src/tbb/arena.cpp:152
#19 0x00007f0c64249ed3 in tbb::internal::market::process (this=0x7f0c60b57580, j=...) at ../../src/tbb/market.cpp:667
#20 0x00007f0c642465cc in tbb::internal::rml::private_worker::run (this=0x7f0c60879080) at ../../src/tbb/private_server.cpp:266
#21 0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#22 0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f0c3abff700 (LWP 316160)):
#0  0x00007f0c62f8ee2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c62f8ecc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c5f114de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c62fc21c9 in syscall () from /lib64/libc.so.6
#5  0x00007f0c642467a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f0c6087902c) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f0c6087902c) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f0c60879020) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f0c60879000) at ../../src/tbb/private_server.cpp:273
#9  0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f0c3bdfe700 (LWP 316156)):
#0  0x00007f0c62f8ee2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c62f8ecc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c5f114de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c62fc21c9 in syscall () from /lib64/libc.so.6
#5  0x00007f0c642467a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f0c608791ac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f0c608791ac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f0c608791a0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f0c60879180) at ../../src/tbb/private_server.cpp:273
#9  0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f0c3c7ff700 (LWP 316153)):
#0  0x00007f0c62f8ee2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c62f8ecc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c5f114de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c62fc21c9 in syscall () from /lib64/libc.so.6
#5  0x00007f0c642467a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f0c6087912c) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f0c6087912c) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f0c60879120) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f0c60879100) at ../../src/tbb/private_server.cpp:273
#9  0x00007f0c642467d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f0c3dbae700 (LWP 316115)):
#0  0x00007f0c62f8ee2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c62f8ecc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c5e9c0869 in evf::FastMonitoringService::dowork() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#3  0x00007f0c6388bccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f0c415ff700 (LWP 316111)):
#0  0x00007f0c632a3965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f0c638865ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc820/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/gcc-8.2.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f0c5e99b519 in FedRawDataInputSource::readWorker(unsigned int) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#4  0x00007f0c6388bccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#5  0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f0c4ced1700 (LWP 316077)):
#0  0x00007f0c632a7179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f0c5f114f97 in edm::service::cmssw_stacktrace_fork() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f0c5f115a5a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f0c6388bccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007f0c6329fdd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f0c62fc7ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f0c614954c0 (LWP 316050)):
#0  0x00007f0c62f8ee2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f0c62f8ecc4 in sleep () from /lib64/libc.so.6
#2  0x00007f0c5f114de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f0c62fc21c9 in syscall () from /lib64/libc.so.6
#5  0x00007f0c64246411 in tbb::internal::futex_wakeup_one (futex=0x7f0c60878f2c) at ../../include/tbb/machine/linux_common.h:90
#6  tbb::internal::binary_semaphore::V (this=0x7f0c60878f2c) at ../../src/tbb/semaphore.h:214
#7  tbb::internal::binary_semaphore::V (this=0x7f0c60878f2c) at ../../src/tbb/semaphore.h:211
#8  rml::internal::thread_monitor::notify (this=0x7f0c60878f20) at ../../src/tbb/../rml/server/thread_monitor.h:240
#9  tbb::internal::rml::private_worker::wake_or_launch (this=0x7f0c60878f00) at ../../src/tbb/private_server.cpp:312
#10 tbb::internal::rml::private_server::wake_some(int) () at ../../src/tbb/private_server.cpp:395
#11 0x00007f0c6424e512 in tbb::internal::generic_scheduler::local_spawn(tbb::task*, tbb::task*&) () at ../../src/tbb/scheduler.cpp:712
#12 0x00007f0c65a6942a in void edm::Worker::doWorkAsync<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f0c65b6e7aa in edm::Path::runNextWorkerAsync(unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007f0c65b6fb64 in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007f0c65b6fc8c in edm::FunctorWaitingTask<edm::Path::runNextWorkerAsync(unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#16 0x00007f0c6425127d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f0c60b44e00, context_guard=..., t=t@entry=0x7f0c3d31bc40, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#17 0x00007f0c64251575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f0c60b44e00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#18 0x00007f0c65b13b85 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#19 0x00007f0c65b1b4de in edm::EventProcessor::runToCompletion() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#20 0x0000000000410c7b in main::{lambda()#1}::operator()() const ()
#21 0x000000000040f2e2 in main ()

Current Modules:

Module: L1TRawToDigi:hltGtStage2Digis (crashed)
Module: none
Module: none
Module: none
Module: EvFOutputModule:hltOutputPhysicsVirginRaw8
Module: none
Module: none
Module: none

[Dr15Jones]

@dan131riley can you think of a good way to figure out why TBB is starting more threads?

[smorovic]

I think that It depends how many events process saw in a LS. I guess if it's not multiple of 8 or 16 it will cause stalling because of waiting for others to finish LS (still due to limitation of one per LS currently).

There indeed seemed to be some overhead when we were running during MWGR #1 (we sample states per CMSSW stream and inter-module transition appeared to be around 10% in the graph). When I ran with 1-thread/stream it disappeared.

This HLT menu is more on the extreme side, designed for the test, in that all work is in output modules. There will be more reconstruction in real Heavy Ion menu and output will not be beyond 40% of CPU budget (in the worst case).

Finally I did get one abort even without hlTFEDSelector, but it definitely takes longer. It looks the same (and has 19 threads).

[Dr15Jones]

I think that It depends how many events process saw in a LS. I guess if it's not multiple of 8 or 16 it will cause stalling because of waiting for others to finish LS (still due to limitation of one per LS currently).

The stall grapher, if I remember correctly how we designed it, would not account for stalls from LS in that way. So this is strictly a case of too many events wanting to run that particular module at any given time and the framework not being able to do it since it is a one module and there are no other modules around to run.

Finally I did get one abort even without hlTFEDSelector, but it definitely takes longer. It looks the same (and has 19 threads).

Please go ahead and post that one as well. The more we have (even if they look similar) the more data to help diagnose what is happening.

[smorovic]

I didn't save it, but I will run again to get it for that case.

On a second thought, EoL waiting case is specially marked also with FastMonitoringService sampling (and would be seen as different contribution), so maybe we were seeing the same thing. I will put it in my plans to investigate under which condition it happens (e.g. if it changes with more tbb threads assigned with same number of streams etc).

[smorovic]

cmsRun: /tmp/smorovic/CMSSW_11_0_1/src/EventFilter/L1TRawToDigi/plugins/L1TRawToDigi.cc:141: virtual void l1t::L1TRawToDigi::produce(edm::Event&, const edm::EventSetup&): Assertion `0' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Mon Mar 23 20:12:08 CET 2020
Thread 12 (Thread 0x7fca33bff700 (LWP 329040)):
#0  0x00007fcac65e2d12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fcac1cd99aa in FedRawDataInputSource::readSupervisor() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#2  0x00007fcac6bcaccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#3  0x00007fcac65dedd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007fcac6306ead in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fca9adfe700 (LWP 328987)):
#0  0x00007fcac62cde2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fcac62cdcc4 in sleep () from /lib64/libc.so.6
#2  0x00007fcac244fde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fcac62ebd47 in sched_yield () from /lib64/libc.so.6
#5  0x00007fcac758f465 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7fcac3c93e00, completion_ref_count=<optimized out>, isolation=0) at ../../src/tbb/mailbox.h:214
#6  0x00007fcac758a2be in tbb::internal::arena::process (this=0x7fcac3dc3480, s=...) at ../../include/tbb/task.h:992
#7  0x00007fcac7588ed3 in tbb::internal::market::process (this=0x7fcac3de3580, j=...) at ../../src/tbb/market.cpp:667
#8  0x00007fcac75855cc in tbb::internal::rml::private_worker::run (this=0x7fcac3b06e80) at ../../src/tbb/private_server.cpp:266
#9  0x00007fcac75857d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007fcac65dedd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fcac6306ead in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fca9b7ff700 (LWP 328985)):
#0  0x00007fcac62cde2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fcac62cdcc4 in sleep () from /lib64/libc.so.6
#2  0x00007fcac244fde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fcac62ebd47 in sched_yield () from /lib64/libc.so.6
#5  0x00007fcac758f465 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7fcac3ca3e00, completion_ref_count=<optimized out>, isolation=0) at ../../src/tbb/mailbox.h:214
#6  0x00007fcac758a2be in tbb::internal::arena::process (this=0x7fcac3dc3480, s=...) at ../../include/tbb/task.h:992
#7  0x00007fcac7588ed3 in tbb::internal::market::process (this=0x7fcac3de3580, j=...) at ../../src/tbb/market.cpp:667
#8  0x00007fcac75855cc in tbb::internal::rml::private_worker::run (this=0x7fcac3b06f80) at ../../src/tbb/private_server.cpp:266
#9  0x00007fcac75857d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007fcac65dedd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fcac6306ead in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fca9d3fe700 (LWP 328980)):
#0  0x00007fcac62cde2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fcac62cdcc4 in sleep () from /lib64/libc.so.6
#2  0x00007fcac244fde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fcac7522e5a in ?? () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#5  0x00007fcac7524962 in ?? () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#6  0x00007fcac7525532 in deflate () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#7  0x00007fcac752e755 in compress2 () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#8  0x00007fcac1c23c9e in edm::StreamSerializer::compressBuffer(unsigned char*, unsigned int, std::vector<unsigned char, std::allocator<unsigned char> >&, int, unsigned int) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#9  0x00007fcac1c25e0b in edm::StreamSerializer::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Hash<1> const&, edm::StreamerCompressionAlgo, int, unsigned int) const () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#10 0x00007fcac1c31c13 in edm::StreamerOutputModuleCommon::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Handle<edm::TriggerResults> const&, edm::Hash<1> const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#11 0x00007fcac1ce365b in evf::EvFOutputModule::write(edm::EventForOutput const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#12 0x00007fcac8edcc9f in edm::one::OutputModuleBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007fcac8e00543 in edm::WorkerT<edm::one::OutputModuleBase>::implDo(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007fcac8da8c8a in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007fcac8da8e5e in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#16 0x00007fcac8daa65b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#17 0x00007fcac8dabcc5 in void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#18 0x00007fcac8dabd81 in edm::SerialTaskQueue::QueuedTask<void edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#19 0x00007fcac759027d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7fcac3cb3e00, context_guard=..., t=0x7fcac3c9d740, t@entry=0x7fcac3cb6040, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#20 0x00007fcac7590575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fcac3cb3e00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#21 0x00007fcac758a35f in tbb::internal::arena::process (this=0x7fcac3dc3480, s=...) at ../../src/tbb/arena.cpp:152
#22 0x00007fcac7588ed3 in tbb::internal::market::process (this=0x7fcac3de3580, j=...) at ../../src/tbb/market.cpp:667
#23 0x00007fcac75855cc in tbb::internal::rml::private_worker::run (this=0x7fcac3b06f00) at ../../src/tbb/private_server.cpp:266
#24 0x00007fcac75857d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#25 0x00007fcac65dedd5 in start_thread () from /lib64/libpthread.so.0
#26 0x00007fcac6306ead in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fca9c9fd700 (LWP 328976)):
#0  0x00007fcac62cde2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fcac62cdcc4 in sleep () from /lib64/libc.so.6
#2  0x00007fcac244fde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fcac758f352 in __TBB_machine_pause (delay=4) at ../../include/tbb/machine/gcc_ia32_common.h:46
#5  tbb::internal::atomic_backoff::bounded_pause (this=<synthetic pointer>) at ../../include/tbb/tbb_machine.h:373
#6  tbb::internal::prolonged_pause () at ../../src/tbb/scheduler_common.h:306
#7  tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7fcac3cbbe00, completion_ref_count=<optimized out>, isolation=0) at ../../src/tbb/custom_scheduler.h:305
#8  0x00007fcac758a2be in tbb::internal::arena::process (this=0x7fcac3dc3480, s=...) at ../../include/tbb/task.h:992
#9  0x00007fcac7588ed3 in tbb::internal::market::process (this=0x7fcac3de3580, j=...) at ../../src/tbb/market.cpp:667
#10 0x00007fcac75855cc in tbb::internal::rml::private_worker::run (this=0x7fcac3b07080) at ../../src/tbb/private_server.cpp:266
#11 0x00007fcac75857d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#12 0x00007fcac65dedd5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007fcac6306ead in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fca9dfff700 (LWP 328973)):
#0  0x00007fcac62cde2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fcac62cdcc4 in sleep () from /lib64/libc.so.6
#2  0x00007fcac244fde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fcac63011c9 in syscall () from /lib64/libc.so.6
#5  0x00007fcac7585411 in tbb::internal::futex_wakeup_one (futex=0x7fcac3b06fac) at ../../include/tbb/machine/linux_common.h:90
#6  tbb::internal::binary_semaphore::V (this=0x7fcac3b06fac) at ../../src/tbb/semaphore.h:214
#7  tbb::internal::binary_semaphore::V (this=0x7fcac3b06fac) at ../../src/tbb/semaphore.h:211
#8  rml::internal::thread_monitor::notify (this=0x7fcac3b06fa0) at ../../src/tbb/../rml/server/thread_monitor.h:240
#9  tbb::internal::rml::private_worker::wake_or_launch (this=0x7fcac3b06f80) at ../../src/tbb/private_server.cpp:312
#10 tbb::internal::rml::private_server::wake_some(int) () at ../../src/tbb/private_server.cpp:395
#11 0x00007fcac758d512 in tbb::internal::generic_scheduler::local_spawn(tbb::task*, tbb::task*&) () at ../../src/tbb/scheduler.cpp:712
#12 0x00007fcac8da842a in void edm::Worker::doWorkAsync<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007fcac8ead7aa in edm::Path::runNextWorkerAsync(unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007fcac8eaeb64 in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007fcac8eaec8c in edm::FunctorWaitingTask<edm::Path::runNextWorkerAsync(unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#16 0x00007fcac759027d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7fcac3c9be00, context_guard=..., t=t@entry=0x7fca9e7ca940, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#17 0x00007fcac7590575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fcac3c9be00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#18 0x00007fcac758a35f in tbb::internal::arena::process (this=0x7fcac3dc3480, s=...) at ../../src/tbb/arena.cpp:152
#19 0x00007fcac7588ed3 in tbb::internal::market::process (this=0x7fcac3de3580, j=...) at ../../src/tbb/market.cpp:667
#20 0x00007fcac75855cc in tbb::internal::rml::private_worker::run (this=0x7fcac3b07000) at ../../src/tbb/private_server.cpp:266
#21 0x00007fcac75857d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#22 0x00007fcac65dedd5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fcac6306ead in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fca9f1fe700 (LWP 328971)):
#0  0x00007fcac62fc20d in poll () from /lib64/libc.so.6
#1  0x00007fcac24503ff in full_read.constprop () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007fcac2450b3c in edm::service::InitRootHandlers::stacktraceFromThread() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007fcac2451a19 in sig_dostack_then_abort () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007fcac623f207 in raise () from /lib64/libc.so.6
#6  0x00007fcac62408f8 in abort () from /lib64/libc.so.6
#7  0x00007fcac6238026 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007fcac62380d2 in __assert_fail () from /lib64/libc.so.6
#9  0x00007fcaa21a30c7 in l1t::L1TRawToDigi::produce(edm::Event&, edm::EventSetup const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginEventFilterL1TRawToDigiAuto.so
#10 0x00007fcac8edf956 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007fcac8e00b83 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007fcac8da8c8a in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007fcac8da8e5e in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007fcac8daa65b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007fcac8dab674 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#16 0x00007fcac759027d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7fcac3cabe00, context_guard=..., t=t@entry=0x7fca9e7cbe40, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#17 0x00007fcac7590575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fcac3cabe00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#18 0x00007fcac758a35f in tbb::internal::arena::process (this=0x7fcac3dc3480, s=...) at ../../src/tbb/arena.cpp:152
#19 0x00007fcac7588ed3 in tbb::internal::market::process (this=0x7fcac3de3580, j=...) at ../../src/tbb/market.cpp:667
#20 0x00007fcac75855cc in tbb::internal::rml::private_worker::run (this=0x7fcac3b07180) at ../../src/tbb/private_server.cpp:266
#21 0x00007fcac75857d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#22 0x00007fcac65dedd5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007fcac6306ead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fca9fbff700 (LWP 328967)):
#0  0x00007fcac62cde2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fcac62cdcc4 in sleep () from /lib64/libc.so.6
#2  0x00007fcac244fde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fcac62ebd47 in sched_yield () from /lib64/libc.so.6
#5  0x00007fcac758f465 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7fcac3cc3e00, completion_ref_count=<optimized out>, isolation=0) at ../../src/tbb/mailbox.h:214
#6  0x00007fcac758a2be in tbb::internal::arena::process (this=0x7fcac3dc3480, s=...) at ../../include/tbb/task.h:992
#7  0x00007fcac7588ed3 in tbb::internal::market::process (this=0x7fcac3de3580, j=...) at ../../src/tbb/market.cpp:667
#8  0x00007fcac75855cc in tbb::internal::rml::private_worker::run (this=0x7fcac3b07100) at ../../src/tbb/private_server.cpp:266
#9  0x00007fcac75857d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007fcac65dedd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fcac6306ead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fcaa08ca700 (LWP 328863)):
#0  0x00007fcac62cde2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fcac62cdcc4 in sleep () from /lib64/libc.so.6
#2  0x00007fcac1cfb869 in evf::FastMonitoringService::dowork() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#3  0x00007fcac6bcaccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007fcac65dedd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fcac6306ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fcaa43ff700 (LWP 328860)):
#0  0x00007fcac65e2965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fcac6bc55ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc820/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/gcc-8.2.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007fcac1cd6519 in FedRawDataInputSource::readWorker(unsigned int) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#4  0x00007fcac6bcaccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#5  0x00007fcac65dedd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007fcac6306ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fcaafbff700 (LWP 328827)):
#0  0x00007fcac65e6179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007fcac244ff97 in edm::service::cmssw_stacktrace_fork() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007fcac2450a5a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007fcac6bcaccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007fcac65dedd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007fcac6306ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fcac47d44c0 (LWP 328798)):
#0  0x00007fcac62cde2d in nanosleep () from /lib64/libc.so.6
#1  0x00007fcac62cdcc4 in sleep () from /lib64/libc.so.6
#2  0x00007fcac244fde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007fcac62ebd47 in sched_yield () from /lib64/libc.so.6
#5  0x00007fcac758f465 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7fcac3dd0e00, completion_ref_count=<optimized out>, isolation=0) at ../../src/tbb/mailbox.h:214
#6  0x00007fcac75907fe in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7fcac3dd0e00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#7  0x00007fcac8e52b85 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#8  0x00007fcac8e5a4de in edm::EventProcessor::runToCompletion() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#9  0x0000000000410c7b in main::{lambda()#1}::operator()() const ()
#10 0x000000000040f2e2 in main ()

Current Modules:

Module: L1TRawToDigi:hltGtStage2Digis (crashed)
Module: none
Module: none
Module: none
Module: none
Module: none
Module: EvFOutputModule:hltOutputPhysicsVirginRaw3
Module: none

[smorovic]

above was actually with full HLTriggerFinalPath

[smorovic]

This one is again without that path and without hltFEDSelector

cmsRun: /tmp/smorovic/CMSSW_11_0_1/src/EventFilter/L1TRawToDigi/plugins/L1TRawToDigi.cc:141: virtual void l1t::L1TRawToDigi::produce(edm::Event&, const edm::EventSetup&): Assertion `0' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Mon Mar 23 20:19:52 CET 2020
Thread 14 (Thread 0x7f1b1e27e700 (LWP 330020)):
#0  0x00007f1bb255ae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1bb255acc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1bae6e1de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1bb258e1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f1bb38127a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f1bafe78e2c) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f1bafe78e2c) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f1bafe78e20) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f1bafe78e00) at ../../src/tbb/private_server.cpp:273
#9  0x00007f1bb38127d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f1b1ec7f700 (LWP 330019)):
#0  0x00007f1bb258920d in poll () from /lib64/libc.so.6
#1  0x00007f1bae6e23ff in full_read.constprop () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f1bae6e2b3c in edm::service::InitRootHandlers::stacktraceFromThread() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f1bae6e3a19 in sig_dostack_then_abort () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f1bb24cc207 in raise () from /lib64/libc.so.6
#6  0x00007f1bb24cd8f8 in abort () from /lib64/libc.so.6
#7  0x00007f1bb24c5026 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007f1bb24c50d2 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f1b8ea2f0c7 in l1t::L1TRawToDigi::produce(edm::Event&, edm::EventSetup const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginEventFilterL1TRawToDigiAuto.so
#10 0x00007f1bb516c956 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#11 0x00007f1bb508db83 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ModuleCallingContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#12 0x00007f1bb5035c8a in decltype ({parm#1}()) edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f1bb5035e5e in bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007f1bb503765b in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr const*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007f1bb5038674 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#16 0x00007f1bb381d27d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f1b8c8cfe00, context_guard=..., t=t@entry=0x7f1bb006eb40, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#17 0x00007f1bb381d575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f1b8c8cfe00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#18 0x00007f1bb381735f in tbb::internal::arena::process (this=0x7f1bb0137480, s=...) at ../../src/tbb/arena.cpp:152
#19 0x00007f1bb3815ed3 in tbb::internal::market::process (this=0x7f1bb0157580, j=...) at ../../src/tbb/market.cpp:667
#20 0x00007f1bb38125cc in tbb::internal::rml::private_worker::run (this=0x7f1bafe78d80) at ../../src/tbb/private_server.cpp:266
#21 0x00007f1bb38127d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#22 0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f1b2237f700 (LWP 330008)):
#0  0x00007f1bb286fd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f1badf6b9aa in FedRawDataInputSource::readSupervisor() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#2  0x00007f1bb2e57ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#3  0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f1b877fc700 (LWP 329999)):
#0  0x00007f1bb255ae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1bb255acc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1bae6e1de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1bb258e1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f1bb38127a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f1bafe78fac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f1bafe78fac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f1bafe78fa0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f1bafe78f80) at ../../src/tbb/private_server.cpp:273
#9  0x00007f1bb38127d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f1b881fd700 (LWP 329996)):
#0  0x00007f1bb255ae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1bb255acc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1bae6e1de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1bb258e1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f1bb38127a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f1bafe78eac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f1bafe78eac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f1bafe78ea0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f1bafe78e80) at ../../src/tbb/private_server.cpp:273
#9  0x00007f1bb38127d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f1b88bfe700 (LWP 329995)):
#0  0x00007f1bb255ae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1bb255acc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1bae6e1de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1bb258e1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f1bb38127a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f1bafe78f2c) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f1bafe78f2c) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f1bafe78f20) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f1bafe78f00) at ../../src/tbb/private_server.cpp:273
#9  0x00007f1bb38127d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f1b897ff700 (LWP 329990)):
#0  0x00007f1bb255ae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1bb255acc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1bae6e1de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1bb258e1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f1bb38127a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f1bafe790ac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f1bafe790ac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f1bafe790a0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f1bafe79080) at ../../src/tbb/private_server.cpp:273
#9  0x00007f1bb38127d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f1b8a9ff700 (LWP 329985)):
#0  0x00007f1bb255ae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1bb255acc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1bae6e1de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1bb2578d47 in sched_yield () from /lib64/libc.so.6
#5  0x00007f1bb381c465 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::receive_or_steal_task (this=0x7f1bb001fe00, completion_ref_count=<optimized out>, isolation=0) at ../../src/tbb/mailbox.h:214
#6  0x00007f1bb38172be in tbb::internal::arena::process (this=0x7f1bb0137480, s=...) at ../../include/tbb/task.h:992
#7  0x00007f1bb3815ed3 in tbb::internal::market::process (this=0x7f1bb0157580, j=...) at ../../src/tbb/market.cpp:667
#8  0x00007f1bb38125cc in tbb::internal::rml::private_worker::run (this=0x7f1bafe79000) at ../../src/tbb/private_server.cpp:266
#9  0x00007f1bb38127d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f1b8b9fe700 (LWP 329978)):
#0  0x00007f1bb255ae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1bb255acc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1bae6e1de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1bb258e1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f1bb38127a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f1bafe791ac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f1bafe791ac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f1bafe791a0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f1bafe79180) at ../../src/tbb/private_server.cpp:273
#9  0x00007f1bb38127d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f1b8c3ff700 (LWP 329977)):
#0  0x00007f1bb255ae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1bb255acc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1bae6e1de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1bb258e1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f1bb38127a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f1bafe7912c) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f1bafe7912c) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f1bafe79120) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f1bafe79100) at ../../src/tbb/private_server.cpp:273
#9  0x00007f1bb38127d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f1b8d17a700 (LWP 329969)):
#0  0x00007f1bb255ae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1bb255acc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1badf8d869 in evf::FastMonitoringService::dowork() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#3  0x00007f1bb2e57ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f1b90bff700 (LWP 329963)):
#0  0x00007f1bb286f965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f1bb2e525ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc820/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/gcc-8.2.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f1badf68519 in FedRawDataInputSource::readWorker(unsigned int) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#4  0x00007f1bb2e57ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#5  0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f1b9c49d700 (LWP 329906)):
#0  0x00007f1bb2873179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f1bae6e1f97 in edm::service::cmssw_stacktrace_fork() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f1bae6e2a5a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f1bb2e57ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007f1bb286bdd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f1bb2593ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f1bb0a614c0 (LWP 329877)):
#0  0x00007f1bb255ae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f1bb255acc4 in sleep () from /lib64/libc.so.6
#2  0x00007f1bae6e1de0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f1bb258e1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f1bb3812411 in tbb::internal::futex_wakeup_one (futex=0x7f1bafe78dac) at ../../include/tbb/machine/linux_common.h:90
#6  tbb::internal::binary_semaphore::V (this=0x7f1bafe78dac) at ../../src/tbb/semaphore.h:214
#7  tbb::internal::binary_semaphore::V (this=0x7f1bafe78dac) at ../../src/tbb/semaphore.h:211
#8  rml::internal::thread_monitor::notify (this=0x7f1bafe78da0) at ../../src/tbb/../rml/server/thread_monitor.h:240
#9  tbb::internal::rml::private_worker::wake_or_launch (this=0x7f1bafe78d80) at ../../src/tbb/private_server.cpp:312
#10 tbb::internal::rml::private_server::wake_some(int) () at ../../src/tbb/private_server.cpp:395
#11 0x00007f1bb381a512 in tbb::internal::generic_scheduler::local_spawn(tbb::task*, tbb::task*&) () at ../../src/tbb/scheduler.cpp:712
#12 0x00007f1bb503542a in void edm::Worker::doWorkAsync<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f1bb513a7aa in edm::Path::runNextWorkerAsync(unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007f1bb513bb64 in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007f1bb513bc8c in edm::FunctorWaitingTask<edm::Path::runNextWorkerAsync(unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#16 0x00007f1bb381d27d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f1bb0144e00, context_guard=..., t=t@entry=0x7f1bb0002740, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#17 0x00007f1bb381d575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f1bb0144e00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#18 0x00007f1bb50dfb85 in edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#19 0x00007f1bb50e74de in edm::EventProcessor::runToCompletion() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#20 0x0000000000410c7b in main::{lambda()#1}::operator()() const ()
#21 0x000000000040f2e2 in main ()

Current Modules:

Module: L1TRawToDigi:hltGtStage2Digis (crashed)
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none
Module: none

A fatal system signal has occurred: abort signal

[Dr15Jones]

Would it be possible in the test to rebuild FWCore/Framework with '-g' option? I.e.

export USER_CXXFLAGS='-g'

? That could give us some of the call parameter values which might help pin things down.

[Dr15Jones]

One striking thing about the tracebacks is how prevalent the following is found in a stack

#12 0x00007f1bb503542a in void edm::Worker::doWorkAsync<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::WaitingTask*, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#13 0x00007f1bb513a7aa in edm::Path::runNextWorkerAsync(unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#14 0x00007f1bb513bb64 in edm::Path::workerFinished(std::__exception_ptr::exception_ptr const*, unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#15 0x00007f1bb513bc8c in edm::FunctorWaitingTask<edm::Path::runNextWorkerAsync(unsigned int, edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ServiceToken const&, edm::StreamID const&, edm::StreamContext const*)::{lambda(std::__exception_ptr::exception_ptr const*)#1}>::execute() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libFWCoreFramework.so

Which is when a Path just finished one module and is starting to span another one.

[smorovic]

Could path spanning prevalence could be related to the stalling problem?

I recompiled FWCore/Framework with 'scram b clean;scram b USER_CXXFLAGS="-g" '. Here is debug stack trace:

cmsRun: /tmp/smorovic/CMSSW_11_0_1/src/EventFilter/L1TRawToDigi/plugins/L1TRawToDigi.cc:141: virtual void l1t::L1TRawToDigi::produce(edm::Event&, const edm::EventSetup&): Assertion `0' failed.

A fatal system signal has occurred: abort signal
The following is the call stack containing the origin of the signal.

Mon Mar 23 20:58:18 CET 2020
Thread 12 (Thread 0x7f970f5ff700 (LWP 346597)):
#0  0x00007f97a12bfd12 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f979c9b59aa in FedRawDataInputSource::readSupervisor() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#2  0x00007f97a18a7ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#3  0x00007f97a12bbdd5 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f97a0fe3ead in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f97759ff700 (LWP 346581)):
#0  0x00007f97a0faae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f97a0faacc4 in sleep () from /lib64/libc.so.6
#2  0x00007f979d12bde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f97a0fde1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f97a22627a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f979e8b6eac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f979e8b6eac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f979e8b6ea0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f979e8b6e80) at ../../src/tbb/private_server.cpp:273
#9  0x00007f97a22627d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f97a12bbdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f97a0fe3ead in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f97769fc700 (LWP 346574)):
#0  0x00007f97a0faae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f97a0faacc4 in sleep () from /lib64/libc.so.6
#2  0x00007f979d12bde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f97a0fde1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f97a22627a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f979e8b6fac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f979e8b6fac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f979e8b6fa0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f979e8b6f80) at ../../src/tbb/private_server.cpp:273
#9  0x00007f97a22627d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f97a12bbdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f97a0fe3ead in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f97775fd700 (LWP 346569)):
#0  0x00007f97a0fd920d in poll () from /lib64/libc.so.6
#1  0x00007f979d12c3ff in full_read.constprop () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f979d12cb3c in edm::service::InitRootHandlers::stacktraceFromThread() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f979d12da19 in sig_dostack_then_abort () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x00007f97a0f1c207 in raise () from /lib64/libc.so.6
#6  0x00007f97a0f1d8f8 in abort () from /lib64/libc.so.6
#7  0x00007f97a0f15026 in __assert_fail_base () from /lib64/libc.so.6
#8  0x00007f97a0f150d2 in __assert_fail () from /lib64/libc.so.6
#9  0x00007f977ce7c0c7 in l1t::L1TRawToDigi::produce(edm::Event&, edm::EventSetup const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginEventFilterL1TRawToDigiAuto.so
#10 0x00007f97a3bbc956 in edm::stream::EDProducerAdaptorBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/stream/EDProducerAdaptorBase.cc:58
#11 0x00007f97a3addb83 in edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo (this=0x7f9786b88d00, ep=..., c=..., mcc=0x7f9786b88d28) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:284
#12 0x00007f97a3a85c8a in edm::workerhelper::CallImpl<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::call (mcc=0x7f9786b88d28, es=..., ep=..., iWorker=0x7f9786b88d00) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:661
#13 bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const (this=<optimized out>) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:1102
#14 edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (iFunc=...) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Utilities/interface/ConvertException.h:20
#15 0x00007f97a3a85e5e in edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> > (this=this@entry=0x7f9786b88d00, ep=..., es=..., streamID=streamID@entry=..., parentContext=..., context=<optimized out>, context@entry=0x7f9786a4a288) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/bits/atomic_base.h:512
#16 0x00007f97a3a8765b in edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> > (this=0x7f9786b88d00, iEPtr=iEPtr@entry=0x0, ep=..., es=..., streamID=..., parentContext=..., context=0x7f9786a4a288) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/atomic:238
#17 0x00007f97a3a88674 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute (this=<optimized out>) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/bits/atomic_base.h:707
#18 0x00007f97a226d27d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f979ea7be00, context_guard=..., t=t@entry=0x7f979e9f7f40, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#19 0x00007f97a226d575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f979ea7be00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#20 0x00007f97a226735f in tbb::internal::arena::process (this=0x7f979eb73480, s=...) at ../../src/tbb/arena.cpp:152
#21 0x00007f97a2265ed3 in tbb::internal::market::process (this=0x7f979eb93580, j=...) at ../../src/tbb/market.cpp:667
#22 0x00007f97a22625cc in tbb::internal::rml::private_worker::run (this=0x7f979e8b7080) at ../../src/tbb/private_server.cpp:266
#23 0x00007f97a22627d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#24 0x00007f97a12bbdd5 in start_thread () from /lib64/libpthread.so.0
#25 0x00007f97a0fe3ead in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f9777ffe700 (LWP 346566)):
#0  0x00007f97a0faae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f97a0faacc4 in sleep () from /lib64/libc.so.6
#2  0x00007f979d12bde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f97a21ffe5a in ?? () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#5  0x00007f97a2201962 in ?? () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#6  0x00007f97a2202532 in deflate () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#7  0x00007f97a220b755 in compress2 () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#8  0x00007f979c8ffc9e in edm::StreamSerializer::compressBuffer(unsigned char*, unsigned int, std::vector<unsigned char, std::allocator<unsigned char> >&, int, unsigned int) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#9  0x00007f979c901e0b in edm::StreamSerializer::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Hash<1> const&, edm::StreamerCompressionAlgo, int, unsigned int) const () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#10 0x00007f979c90dc13 in edm::StreamerOutputModuleCommon::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Handle<edm::TriggerResults> const&, edm::Hash<1> const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#11 0x00007f979c9bf65b in evf::EvFOutputModule::write(edm::EventForOutput const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#12 0x00007f97a3bb9c9f in edm::one::OutputModuleBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/one/OutputModuleBase.cc:246
#13 0x00007f97a3add543 in edm::WorkerT<edm::one::OutputModuleBase>::implDo (this=0x7f9786b5df40, ep=..., c=..., mcc=0x7f9786b5df68) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:284
#14 0x00007f97a3a85c8a in edm::workerhelper::CallImpl<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::call (mcc=0x7f9786b5df68, es=..., ep=..., iWorker=0x7f9786b5df40) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:661
#15 bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const (this=<optimized out>) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:1102
#16 edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (iFunc=...) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Utilities/interface/ConvertException.h:20
#17 0x00007f97a3a85e5e in edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> > (this=this@entry=0x7f9786b5df40, ep=..., es=..., streamID=streamID@entry=..., parentContext=..., context=<optimized out>, context@entry=0x7f97868a3888) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/bits/atomic_base.h:512
#18 0x00007f97a3a8765b in edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> > (this=0x7f9786b5df40, iEPtr=iEPtr@entry=0x0, ep=..., es=..., streamID=..., parentContext=..., context=0x7f97868a3888) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/atomic:238
#19 0x00007f97a3a88cc5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}::operator()() const (this=0x7f977acdbe58) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:667
#20 edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&) (this=0x7f9786b1a298, iAction=...) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueueChain.h:150
#21 0x00007f97a3a88d81 in void edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}::operator()() (this=<optimized out>) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueueChain.h:88
#22 edm::SerialTaskQueue::QueuedTask<void edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (this=0x7f977acdbe40) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueue.h:213
#23 0x00007f97a226d27d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f979ea83e00, context_guard=..., t=t@entry=0x7f977acdbe40, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#24 0x00007f97a226d575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f979ea83e00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#25 0x00007f97a226735f in tbb::internal::arena::process (this=0x7f979eb73480, s=...) at ../../src/tbb/arena.cpp:152
#26 0x00007f97a2265ed3 in tbb::internal::market::process (this=0x7f979eb93580, j=...) at ../../src/tbb/market.cpp:667
#27 0x00007f97a22625cc in tbb::internal::rml::private_worker::run (this=0x7f979e8b6f00) at ../../src/tbb/private_server.cpp:266
#28 0x00007f97a22627d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#29 0x00007f97a12bbdd5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007f97a0fe3ead in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f9778bff700 (LWP 346563)):
#0  0x00007f97a0faae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f97a0faacc4 in sleep () from /lib64/libc.so.6
#2  0x00007f979d12bde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f97a21ffe5a in ?? () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#5  0x00007f97a2201962 in ?? () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#6  0x00007f97a2202532 in deflate () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#7  0x00007f97a220b755 in compress2 () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#8  0x00007f979c8ffc9e in edm::StreamSerializer::compressBuffer(unsigned char*, unsigned int, std::vector<unsigned char, std::allocator<unsigned char> >&, int, unsigned int) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#9  0x00007f979c901e0b in edm::StreamSerializer::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Hash<1> const&, edm::StreamerCompressionAlgo, int, unsigned int) const () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#10 0x00007f979c90dc13 in edm::StreamerOutputModuleCommon::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Handle<edm::TriggerResults> const&, edm::Hash<1> const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#11 0x00007f979c9bf65b in evf::EvFOutputModule::write(edm::EventForOutput const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#12 0x00007f97a3bb9c9f in edm::one::OutputModuleBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/one/OutputModuleBase.cc:246
#13 0x00007f97a3add543 in edm::WorkerT<edm::one::OutputModuleBase>::implDo (this=0x7f9786b71540, ep=..., c=..., mcc=0x7f9786b71568) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:284
#14 0x00007f97a3a85c8a in edm::workerhelper::CallImpl<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::call (mcc=0x7f9786b71568, es=..., ep=..., iWorker=0x7f9786b71540) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:661
#15 bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const (this=<optimized out>) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:1102
#16 edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (iFunc=...) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Utilities/interface/ConvertException.h:20
#17 0x00007f97a3a85e5e in edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> > (this=this@entry=0x7f9786b71540, ep=..., es=..., streamID=streamID@entry=..., parentContext=..., context=<optimized out>, context@entry=0x7f9786a49d88) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/bits/atomic_base.h:512
#18 0x00007f97a3a8765b in edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> > (this=0x7f9786b71540, iEPtr=iEPtr@entry=0x0, ep=..., es=..., streamID=..., parentContext=..., context=0x7f9786a49d88) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/atomic:238
#19 0x00007f97a3a88cc5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}::operator()() const (this=0x7f977ad2ba58) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:667
#20 edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&) (this=0x7f9786b4a498, iAction=...) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueueChain.h:150
#21 0x00007f97a3a88d81 in void edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}::operator()() (this=<optimized out>) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueueChain.h:88
#22 edm::SerialTaskQueue::QueuedTask<void edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (this=0x7f977ad2ba40) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueue.h:213
#23 0x00007f97a226d27d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f979ea2be00, context_guard=..., t=0x7f977ad2ba40, t@entry=0x7f979e9d6840, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#24 0x00007f97a226d575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f979ea2be00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#25 0x00007f97a226735f in tbb::internal::arena::process (this=0x7f979eb73480, s=...) at ../../src/tbb/arena.cpp:152
#26 0x00007f97a2265ed3 in tbb::internal::market::process (this=0x7f979eb93580, j=...) at ../../src/tbb/market.cpp:667
#27 0x00007f97a22625cc in tbb::internal::rml::private_worker::run (this=0x7f979e8b7000) at ../../src/tbb/private_server.cpp:266
#28 0x00007f97a22627d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#29 0x00007f97a12bbdd5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007f97a0fe3ead in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f9779dfe700 (LWP 346561)):
#0  0x00007f97a0faae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f97a0faacc4 in sleep () from /lib64/libc.so.6
#2  0x00007f979d12bde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f97a0fde1c9 in syscall () from /lib64/libc.so.6
#5  0x00007f97a22627a5 in tbb::internal::futex_wait (comparand=2, futex=0x7f979e8b71ac) at ../../include/tbb/machine/linux_common.h:81
#6  tbb::internal::binary_semaphore::P (this=0x7f979e8b71ac) at ../../src/tbb/semaphore.h:205
#7  rml::internal::thread_monitor::commit_wait (c=..., this=0x7f979e8b71a0) at ../../src/tbb/../rml/server/thread_monitor.h:255
#8  tbb::internal::rml::private_worker::run (this=0x7f979e8b7180) at ../../src/tbb/private_server.cpp:273
#9  0x00007f97a22627d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#10 0x00007f97a12bbdd5 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f97a0fe3ead in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f977a7ff700 (LWP 346554)):
#0  0x00007f97a0faae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f97a0faacc4 in sleep () from /lib64/libc.so.6
#2  0x00007f979d12bde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f97a21ffe5a in ?? () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#5  0x00007f97a2201962 in ?? () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#6  0x00007f97a2202532 in deflate () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#7  0x00007f97a220b755 in compress2 () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#8  0x00007f979c8ffc9e in edm::StreamSerializer::compressBuffer(unsigned char*, unsigned int, std::vector<unsigned char, std::allocator<unsigned char> >&, int, unsigned int) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#9  0x00007f979c901e0b in edm::StreamSerializer::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Hash<1> const&, edm::StreamerCompressionAlgo, int, unsigned int) const () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#10 0x00007f979c90dc13 in edm::StreamerOutputModuleCommon::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Handle<edm::TriggerResults> const&, edm::Hash<1> const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#11 0x00007f979c9bf65b in evf::EvFOutputModule::write(edm::EventForOutput const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#12 0x00007f97a3bb9c9f in edm::one::OutputModuleBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/one/OutputModuleBase.cc:246
#13 0x00007f97a3add543 in edm::WorkerT<edm::one::OutputModuleBase>::implDo (this=0x7f9786b87080, ep=..., c=..., mcc=0x7f9786b870a8) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:284
#14 0x00007f97a3a85c8a in edm::workerhelper::CallImpl<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::call (mcc=0x7f9786b870a8, es=..., ep=..., iWorker=0x7f9786b87080) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:661
#15 bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const (this=<optimized out>) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:1102
#16 edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (iFunc=...) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Utilities/interface/ConvertException.h:20
#17 0x00007f97a3a85e5e in edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> > (this=this@entry=0x7f9786b87080, ep=..., es=..., streamID=streamID@entry=..., parentContext=..., context=<optimized out>, context@entry=0x7f9786a4a008) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/bits/atomic_base.h:512
#18 0x00007f97a3a8765b in edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> > (this=0x7f9786b87080, iEPtr=iEPtr@entry=0x0, ep=..., es=..., streamID=..., parentContext=..., context=0x7f9786a4a008) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/atomic:238
#19 0x00007f97a3a88cc5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}::operator()() const (this=0x7f977ad29658) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:667
#20 edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&) (this=0x7f9786b07698, iAction=...) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueueChain.h:150
#21 0x00007f97a3a88d81 in void edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}::operator()() (this=<optimized out>) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueueChain.h:88
#22 edm::SerialTaskQueue::QueuedTask<void edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (this=0x7f977ad29640) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueue.h:213
#23 0x00007f97a226d27d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f979ea73e00, context_guard=..., t=0x7f977ad29640, t@entry=0x7f979ea77140, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#24 0x00007f97a226d575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f979ea73e00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#25 0x00007f97a226735f in tbb::internal::arena::process (this=0x7f979eb73480, s=...) at ../../src/tbb/arena.cpp:152
#26 0x00007f97a2265ed3 in tbb::internal::market::process (this=0x7f979eb93580, j=...) at ../../src/tbb/market.cpp:667
#27 0x00007f97a22625cc in tbb::internal::rml::private_worker::run (this=0x7f979e8b7100) at ../../src/tbb/private_server.cpp:266
#28 0x00007f97a22627d9 in tbb::internal::rml::private_worker::thread_routine (arg=<optimized out>) at ../../src/tbb/private_server.cpp:219
#29 0x00007f97a12bbdd5 in start_thread () from /lib64/libpthread.so.0
#30 0x00007f97a0fe3ead in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f977b5a3700 (LWP 346548)):
#0  0x00007f97a0faae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f97a0faacc4 in sleep () from /lib64/libc.so.6
#2  0x00007f979c9d7869 in evf::FastMonitoringService::dowork() () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#3  0x00007f97a18a7ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007f97a12bbdd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f97a0fe3ead in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f977efff700 (LWP 346546)):
#0  0x00007f97a12bf965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f97a18a25ec in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>) at /build/cmsbld/auto-builds/CMSSW_10_6_0_pre4-slc7_amd64_gcc820/build/CMSSW_10_6_0_pre4-build/BUILD/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/gcc-8.2.0/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...) at ../../../../../libstdc++-v3/src/c++11/condition_variable.cc:53
#3  0x00007f979c9b2519 in FedRawDataInputSource::readWorker(unsigned int) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#4  0x00007f97a18a7ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#5  0x00007f97a12bbdd5 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f97a0fe3ead in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f978aeed700 (LWP 346489)):
#0  0x00007f97a12c3179 in waitpid () from /lib64/libpthread.so.0
#1  0x00007f979d12bf97 in edm::service::cmssw_stacktrace_fork() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007f979d12ca5a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  0x00007f97a18a7ccf in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#4  0x00007f97a12bbdd5 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f97a0fe3ead in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f979f4b14c0 (LWP 346438)):
#0  0x00007f97a0faae2d in nanosleep () from /lib64/libc.so.6
#1  0x00007f97a0faacc4 in sleep () from /lib64/libc.so.6
#2  0x00007f979d12bde0 in sig_pause_for_stacktrace () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x00007f97a21ffe5a in ?? () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#5  0x00007f97a2201962 in ?? () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#6  0x00007f97a2202532 in deflate () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#7  0x00007f97a220b755 in compress2 () from /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/external/slc7_amd64_gcc820/lib/libz.so.1
#8  0x00007f979c8ffc9e in edm::StreamSerializer::compressBuffer(unsigned char*, unsigned int, std::vector<unsigned char, std::allocator<unsigned char> >&, int, unsigned int) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#9  0x00007f979c901e0b in edm::StreamSerializer::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Hash<1> const&, edm::StreamerCompressionAlgo, int, unsigned int) const () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#10 0x00007f979c90dc13 in edm::StreamerOutputModuleCommon::serializeEvent(SerializeDataBuffer&, edm::EventForOutput const&, edm::Handle<edm::TriggerResults> const&, edm::Hash<1> const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libIOPoolStreamer.so
#11 0x00007f979c9bf65b in evf::EvFOutputModule::write(edm::EventForOutput const&) () from /tmp/smorovic/CMSSW_11_0_1/lib/slc7_amd64_gcc820/libEventFilterUtilities.so
#12 0x00007f97a3bb9c9f in edm::one::OutputModuleBase::doEvent(edm::EventPrincipal const&, edm::EventSetupImpl const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*) () at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/one/OutputModuleBase.cc:246
#13 0x00007f97a3add543 in edm::WorkerT<edm::one::OutputModuleBase>::implDo (this=0x7f9786b64080, ep=..., c=..., mcc=0x7f9786b640a8) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:284
#14 0x00007f97a3a85c8a in edm::workerhelper::CallImpl<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::call (mcc=0x7f9786b640a8, es=..., ep=..., iWorker=0x7f9786b64080) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:661
#15 bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}::operator()() const (this=<optimized out>) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:1102
#16 edm::convertException::wrap<bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(bool edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::MyPrincipal const&, edm::EventSetupImpl const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}) (iFunc=...) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Utilities/interface/ConvertException.h:20
#17 0x00007f97a3a85e5e in edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> > (this=this@entry=0x7f9786b64080, ep=..., es=..., streamID=streamID@entry=..., parentContext=..., context=<optimized out>, context@entry=0x7f9786a48e88) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/bits/atomic_base.h:512
#18 0x00007f97a3a8765b in edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> > (this=0x7f9786b64080, iEPtr=iEPtr@entry=0x0, ep=..., es=..., streamID=..., parentContext=..., context=0x7f9786a48e88) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/atomic:238
#19 0x00007f97a3a88cc5 in edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}::operator()() const (this=0x7f979ea85d58) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/Worker.h:667
#20 edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&) (this=0x7f9786b24c98, iAction=...) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueueChain.h:150
#21 0x00007f97a3a88d81 in void edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}::operator()() (this=<optimized out>) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueueChain.h:88
#22 edm::SerialTaskQueue::QueuedTask<void edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (this=0x7f979ea85d40) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Concurrency/interface/SerialTaskQueue.h:213
#23 0x00007f97a226d27d in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::process_bypass_loop (this=this@entry=0x7f979eb80e00, context_guard=..., t=0x7f979ea85d40, t@entry=0x7f979eaa6540, isolation=isolation@entry=0) at ../../include/tbb/task.h:992
#24 0x00007f97a226d575 in tbb::internal::custom_scheduler<tbb::internal::IntelSchedulerTraits>::local_wait_for_all (this=0x7f979eb80e00, parent=..., child=<optimized out>) at ../../include/tbb/task.h:992
#25 0x00007f97a3b2fb85 in tbb::task::wait_for_all (this=0x7f979eaaad40) at /opt/offline/slc7_amd64_gcc820/external/tbb/2019_U9/include/tbb/task.h:992
#26 edm::EventProcessor::processLumis(std::shared_ptr<void> const&) () at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/EventProcessor.cc:1052
#27 0x00007f97a3b374de in edm::(anonymous namespace)::LumisInRunProcessor::processLumis (iRun=..., iEP=..., this=0x7fff99e7e518) at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/ext/atomicity.h:96
#28 edm::(anonymous namespace)::RunsInFileProcessor::processRuns (iEP=..., this=<optimized out>) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/TransitionProcessors.icc:119
#29 edm::(anonymous namespace)::FilesProcessor::processFiles (iEP=..., this=0x7fff99e7e500) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/TransitionProcessors.icc:186
#30 edm::EventProcessor::<lambda()>::operator() (__closure=<optimized out>, __closure=<optimized out>) at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/EventProcessor.cc:700
#31 edm::convertException::wrap<edm::EventProcessor::runToCompletion()::<lambda()> > (iFunc=...) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Utilities/interface/ConvertException.h:20
#32 edm::EventProcessor::runToCompletion() () at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/src/EventProcessor.cc:689
#33 0x0000000000410c7b in main::{lambda()#1}::operator()() const () at /opt/offline/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/include/c++/8.3.1/bits/unique_ptr.h:342
#34 0x000000000040f2e2 in edm::convertException::wrap<main(int, char**)::<lambda()> > (iFunc=...) at /opt/offline/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_0_1/src/FWCore/Utilities/interface/ConvertException.h:18
#35 main () at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/bin/cmsRun.cpp:136
#36 0x00007f97a0f083d5 in __libc_start_main () from /lib64/libc.so.6
#37 0x000000000040f3e8 in _start () at /tmp/smorovic/CMSSW_11_0_1/src/FWCore/Framework/bin/cmsRun.cpp:388

Current Modules:

Module: L1TRawToDigi:hltGtStage2Digis (crashed)
Module: none
Module: EvFOutputModule:hltOutputPhysicsVirginRaw5
Module: EvFOutputModule:hltOutputPhysicsVirginRaw3
Module: EvFOutputModule:hltOutputPhysicsVirginRaw8
Module: none
Module: EvFOutputModule:hltOutputPhysicsVirginRaw4
Module: none

A fatal system signal has occurred: abort signal

[smorovic]

I caught also one case with the Tracer module. Compressed version is here:

/afs/cern.ch/user/s/smorovic/public/HLT-errors/tracer.log.tar.gz

[Dr15Jones]

Could path spanning prevalence could be related to the stalling problem?

The only thing that cause stalls are one and legacy modules. The same module on multiple Paths do not.

[Dr15Jones]

Even with -g to stacktrace doesn't have the value for StreamID which is what I was hoping to see :(.

[Dr15Jones]

The Tracer output is VERY interesting. The assert appears to happen on stream 3. But at the time I see

++++++++ starting: processing event for module: stream = 3 label = 'hltPrePhysicsVirginRaw5Output' id = 69
++++++++ finished: processing event for module: stream = 3 label = 'hltPrePhysicsVirginRaw5Output' id = 69
++++++++ starting: processing event for module: stream = 3 label = 'PhysicsVirginRaw5Output' id = 27
++++++++ finished: processing event for module: stream = 3 label = 'PhysicsVirginRaw5Output' id = 27
++++++ finished: processing path 'PhysicsVirginRaw5Output' : stream = 3
++++ finished: processing event : stream = 3 run = 1000001704 lumi = 18184 event = 6997951 time = 6795469518833113312
++++ starting: end lumi: stream = 3 run = 1000001704 lumi = 18184 time = 6795469518833151744
++++++++ starting: prefetching before processing event for module: stream = 3 label = 'hltGtStage2Digis' id = 32
++++++ starting: end lumi for module: stream = 3 label = 'hltPrePhysicsVirginRaw2Output' id = 56
++++++ finished: end lumi for module: stream = 3 label = 'hltPrePhysicsVirginRaw2Output' id = 56
++++++ starting: end lumi for module: stream = 3 label = 'hltPrePhysicsVirginRaw1Output' id = 54

Note that right after stream 3 starts doing 'end lumi' We get the line

++++++++ starting: prefetching before processing event for module: stream = 3 label = 'hltGtStage2Digis' id = 32

It is possible that the MessageLogger is printing out of order, but I don't think so as later

++++++ starting: end lumi for module: stream = 3 label = 'hltPreRandom6' id = 40
++++++ finished: end lumi for module: stream = 3 label = 'hltPreRandom6' id = 40
++++++++ finished: prefetching before processing event for module: stream = 3 label = 'hltGtStage2Digis' id = 32
++++++++ starting: processing event for module: stream = 3 label = 'hltGtStage2Digis' id = 32
cmsRun: /tmp/smorovic/CMSSW_11_0_1/src/EventFilter/L1TRawToDigi/plugins/L1TRawToDigi.cc:141: virtual void l1t::L1TRawToDigi::produce(edm::Event&, const edm::EventSetup&): Assertion `0' failed.

[Dr15Jones]

So the last event processed by stream 3 before doing the end lumi, all the Paths report finishing properly.

[Dr15Jones]

Here is an interesting discovery about the configuration

process.PhysicsVirginRaw8Output = cms.EndPath( process.hltOutputPhysicsVirginRaw8 + process.hltGtStage2Digis + process.hltPrePhysicsVirginRaw8Output )
process.PhysicsVirginRaw7Output = cms.EndPath( process.hltOutputPhysicsVirginRaw7 + process.hltGtStage2Digis + process.chltPrePhysicsVirginRaw7Output )
process.PhysicsVirginRaw6Output = cms.EndPath( process.hltOutputPhysicsVirginRaw6 + process.hltGtStage2Digis + process.hltPrePhysicsVirginRaw6Output )
process.PhysicsVirginRaw5Output = cms.EndPath( process.hltOutputPhysicsVirginRaw5 + process.hltGtStage2Digis + process.hltPrePhysicsVirginRaw5Output )

Have a dependency inversion. Modules process.hltOutputPhysicsVirginRaw* depend on TriggerSummaryProducerAOD/'hltTriggerSummaryAOD' which depends on hltGtStage2Digis. But as you can see in the above configuration, hltGtStage2Digis comes after process.hltOutputPhysicsVirginRaw*. The only reason this works is several of the other Paths contain hltGtStage2Digis.

I would suggest fixing that inversion for your future testing.

[smorovic]

Thanks, I corrected it and now the same test has been running for an hour without a problem! Just to be sure, I will do a bit more testing and leave it running until tomorrow.

I will also open a ticket with the HLT group to fix this kind of issue in HLT configurations.

[smorovic]

I have been running for several hours in total (it fills disk after 1 hour), but so far no assertion (I would get it within minutes normally). It seems that the problem is solved by proper ordering of the modules.

[Dr15Jones]

Even with a 'solution' I'd like to keep this issue open as we explore the underlying cause.

[smorovic]

I agree, as it is still not understood. I am available to do more tests as much as needed. Also I opened the JIRA ticket with TSG group to raise this issue with the HLT configurations: https://its.cern.ch/jira/browse/CMSHLT-2112

[Martin-Grunewald]

The order should always be: first hltGtStage2Digis, followed by HLTPrescaler module, followed by Output module. The HLTPrescaler consumes something from hltGtStage2Digis, and should itself be before the OutputModule so that the OutputModule does not see all events!

[Martin-Grunewald]

TriggerSummaryProducer runs in the "last" path, HLTriggerFinalPath, after all other paths and before any EndPath!

[Dr15Jones]

@Martin-Grunewald Actually, EndPaths and Paths have been started concurrently for many years now.

[Dr15Jones]

From looking at the Tracer output, we tend to see HLTriggerFinalPath finishing before HLTriggerFirstPath finishes!

[Martin-Grunewald]

Hopefully not at HLT. The TriggerSummary makes sense to construct only if all other paths have given their decision, as the summary records their outcome. I think you have put in safeguards in the summary producer code, no?

GetterOfProducts thingee....

[Dr15Jones]

I have a hypothesis as to what is happening. The way Paths work is when it is time for the Nth module to run

• we create a task which will run the N+1 module
• we pass that task to the Nth module's runModuleAsync call
• the runModuleAsync call adds the task to the list of tasks to run once the module finishes
• then runModuleAsync try to update an atomic value to see if it should actually run the module

Now say we have the same module on M paths. All the paths start at the same time so we could get M different calls to runModuleAsync. Now say one of the threads pauses exactly between adding the task to the list and trying to update the atomic value. Since another task is running the module, eventually all waiting tasks will be run and the Event will finish. Once an Event is finished, we reset the Path so the atomic value gets reset to false. Now lets say the paused thread starts up at this point. It would check the atomic and find it unset so it would create a new task to run the module even though the Event for which the task was meant has finished!

The fix is simple, change the order of trying to update the atomic and adding to the waiting task list.

[makortel]

@Martin-Grunewald

The TriggerSummaryProducerRAW and TriggerSummaryProducerAOD indeed use edm::GetterOfProducts for their input objects. Therefore framework runs those summary producers after all their input objects are either available, or known to not be produced ever for that event because of filters. It may well be that the summary producers (that do not depend on the TriggerResults) are run before the Paths containing the producers of the "input objects" complete.

[Dr15Jones]

@smorovic I think #29291 could fix the issue even with the data dependency inversion problem. Would it be possible for you to run the original test with this change as well? You would have to rebuild all packages containing ED modules you run in your job.

[dan131riley]

One oddity with the traceback with 19 threads is that most of the excess threads aren't in sig_pause_for_stacktrace(), which I think means they started before InitRootHandlers' task_scheduler_observer started and never stole any work, according to the docs at

https://www.threadingbuildingblocks.org/docs/help/reference/task_scheduler/task_scheduler_observer/task_scheduler_observer_member.html

Unfortunately I can't tell from that traceback if all the TBB threads are in the same arena.

I think we've seen cases before where we've somehow ended up with extra TBB threads apparently spontaneously, but this case seems a bit extreme.

-dan

[smorovic]

@smorovic I think #29291 could fix the issue even with the data dependency inversion problem. Would it be possible for you to run the original test with this change as well? You would have to rebuild all packages containing ED modules you run in your job.

Good news. I reverted to the old configuration and with unpatched 11_0_1 got 3 failures in 30 minutes. Then I applied the patch, recompiled all ED modules used and so far have accumulated about 5 hours of running without any error. So, apparently, #29291 fixes it.

[Dr15Jones]

@smorovic thanks so much for running the tests with the code changes!

[smorovic]

No problem, I am glad that we could pinpoint the problem!

[Dr15Jones]

I think we could close this issue now.

[smorovic]

Thanks. Closing the issue.