search

art-daq / artdaq

Other
0 stars 3 forks source link

EventBuilder crash pdune_Aug20A #106

Closed eflumerf closed 2 years ago

eflumerf commented 2 years ago

This issue has been migrated from https://cdcvs.fnal.gov/redmine/issues/20645 (FNAL account required) Originally created by @ron003 on 2018-08-20 17:52:50

Core was generated by `EventBuilderMain -c id: 5236 commanderPluginType: xmlrpc application_name: Even'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fd2a3147b20 in cling::Interpreter::runAndRemoveStaticDestructors() () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
[Current thread is 1 (Thread 0x7fcebb82e700 (LWP 47117))]
(gdb) bt
#0  0x00007fd2a3147b20 in cling::Interpreter::runAndRemoveStaticDestructors() () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#1  0x00007fd2a30efed6 in TCling::ResetGlobals() () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#2  0x00007fd83f6f2953 in TROOT::EndOfProcessCleanups() () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCore.so
#3  0x00007fd83f7f82fd in TUnixSystem::Exit(int, bool) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCore.so
#4  0x00007fd83f7fc0b2 in TUnixSystem::DispatchSignals(ESignals) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCore.so
#5  
#6  0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
#7  0x00007fd530f1fded in artdaq::TCPSocketTransfer::receiveFragmentHeader (this=, header=..., timeout_usec=)
    at /home/ron/work/artdaqPrj/demo25-kurt2/srcs/artdaq/artdaq/TransferPlugins/TCPSocket_transfer.cc:192
#8  0x00007fd83e865da8 in artdaq::DataReceiverManager::runReceiver_ (this=0x7fd52c05aab0, source_rank=) at /home/ron/work/artdaqPrj/demo25-kurt2/srcs/artdaq/artdaq/DAQrate/DataReceiverManager.cc:208
#9  0x00007fd83d3c291d in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:171
#10 0x00007fd83d5dfe25 in start_thread () from /lib64/libpthread.so.0
#11 0x00007fd83b2c534d in clone () from /lib64/libc.so.6
(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0x7fcebb82e700 (LWP 47117) 0x00007fd2a3147b20 in cling::Interpreter::runAndRemoveStaticDestructors() () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
  2    Thread 0x7fcf5e08b700 (LWP 47112) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  3    Thread 0x7fcfe8b84700 (LWP 47031) 0x00007fd83b28bf00 in waitid () from /lib64/libc.so.6
  4    Thread 0x7fcf5e85c700 (LWP 47111) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  5    Thread 0x7fcf5d0e9700 (LWP 47114) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  6    Thread 0x7fcebbfff700 (LWP 47116) 0x00007fd2a35ee450 in llvm::SimpleBitstreamCursor::Read(unsigned int) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
  7    Thread 0x7fceba0bb700 (LWP 47120) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  8    Thread 0x7fceb9119700 (LWP 47122) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  9    Thread 0x7fceba88c700 (LWP 47119) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  10   Thread 0x7fd530ec5700 (LWP 46791) 0x00007fd83b2bc7a3 in select () from /lib64/libc.so.6
  11   Thread 0x7fcf5c918700 (LWP 47115) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  12   Thread 0x7fce9bfff700 (LWP 47124) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  13   Thread 0x7fce9a88c700 (LWP 47127) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  14   Thread 0x7fd840be9f80 (LWP 39033) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  15   Thread 0x7fcfe9355700 (LWP 47028) 0x00007fd83d5e3cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  16   Thread 0x7fceb8948700 (LWP 47123) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  17   Thread 0x7fceb98ea700 (LWP 47121) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  18   Thread 0x7fce9b05d700 (LWP 47126) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  19   Thread 0x7fcf5f7fe700 (LWP 47108) 0x00007fd83b28be09 in waitpid () from /lib64/libc.so.6
  20   Thread 0x7fcf5ffff700 (LWP 47109) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  21   Thread 0x7fce9b82e700 (LWP 47125) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  22   Thread 0x7fcf5f02d700 (LWP 47110) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  23   Thread 0x7fcebb05d700 (LWP 47118) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
  24   Thread 0x7fcf5d8ba700 (LWP 47113) 0x00007fd83b2baa3d in poll () from /lib64/libc.so.6
(gdb) thread 6
[Switching to thread 6 (Thread 0x7fcebbfff700 (LWP 47116))]
#0  0x00007fd2a35ee450 in llvm::SimpleBitstreamCursor::Read(unsigned int) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
(gdb) bt
#0  0x00007fd2a35ee450 in llvm::SimpleBitstreamCursor::Read(unsigned int) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#1  0x00007fd2a3617b8a in clang::ASTReader::ReadLexicalDeclContextStorage(clang::serialization::ModuleFile&, llvm::BitstreamCursor&, unsigned long, clang::DeclContext*) ()
   from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#2  0x00007fd2a3667c8b in clang::ASTReader::ReadDeclRecord(unsigned int) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#3  0x00007fd2a361a6bd in clang::ASTReader::GetDecl(unsigned int) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#4  0x00007fd2a36575c0 in clang::ASTReader::loadPendingDeclChain(clang::Decl*, unsigned long) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#5  0x00007fd2a3632d96 in clang::ASTReader::finishPendingActions() () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#6  0x00007fd2a3634685 in clang::ASTReader::FinishedDeserializing() () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#7  0x00007fd2a3667461 in clang::ASTReader::ReadDeclRecord(unsigned int) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#8  0x00007fd2a361a6bd in clang::ASTReader::GetDecl(unsigned int) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#9  0x00007fd2a4b20903 in clang::ClassTemplateDecl::LoadLazySpecializations() const () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#10 0x00007fd2a35ece33 in clang::ASTReader::CompleteRedeclChain(clang::Decl const*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#11 0x00007fd2a32af388 in isSafeToConvert(clang::RecordDecl const*, clang::CodeGen::CodeGenTypes&, llvm::SmallPtrSet&) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#12 0x00007fd2a32af2fa in isSafeToConvert(clang::RecordDecl const*, clang::CodeGen::CodeGenTypes&, llvm::SmallPtrSet&) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#13 0x00007fd2a32af182 in isSafeToConvert(clang::RecordDecl const*, clang::CodeGen::CodeGenTypes&, llvm::SmallPtrSet&) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#14 0x00007fd2a32af182 in isSafeToConvert(clang::RecordDecl const*, clang::CodeGen::CodeGenTypes&, llvm::SmallPtrSet&) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#15 0x00007fd2a32af182 in isSafeToConvert(clang::RecordDecl const*, clang::CodeGen::CodeGenTypes&, llvm::SmallPtrSet&) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#16 0x00007fd2a32af5c7 in isSafeToConvert(clang::RecordDecl const*, clang::CodeGen::CodeGenTypes&) [clone .part.229] () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#17 0x00007fd2a32af835 in clang::CodeGen::CodeGenTypes::ConvertRecordDeclType(clang::RecordDecl const*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#18 0x00007fd2a32b1b48 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#19 0x00007fd2a32b236f in clang::CodeGen::CodeGenTypes::ConvertTypeForMem(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#20 0x00007fd2a32b1ed5 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#21 0x00007fd2a32b236f in clang::CodeGen::CodeGenTypes::ConvertTypeForMem(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#22 0x00007fd2a322b400 in (anonymous namespace)::CGRecordLowering::lower(bool) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#23 0x00007fd2a322dad7 in clang::CodeGen::CodeGenTypes::ComputeRecordLayout(clang::RecordDecl const*, llvm::StructType*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#24 0x00007fd2a32afa1d in clang::CodeGen::CodeGenTypes::ConvertRecordDeclType(clang::RecordDecl const*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#25 0x00007fd2a32b1b48 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#26 0x00007fd2a32b236f in clang::CodeGen::CodeGenTypes::ConvertTypeForMem(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#27 0x00007fd2a32b1ed5 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#28 0x00007fd2a32b236f in clang::CodeGen::CodeGenTypes::ConvertTypeForMem(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#29 0x00007fd2a322b400 in (anonymous namespace)::CGRecordLowering::lower(bool) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#30 0x00007fd2a322dad7 in clang::CodeGen::CodeGenTypes::ComputeRecordLayout(clang::RecordDecl const*, llvm::StructType*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#31 0x00007fd2a32afa1d in clang::CodeGen::CodeGenTypes::ConvertRecordDeclType(clang::RecordDecl const*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#32 0x00007fd2a32b1b48 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#33 0x00007fd2a32b236f in clang::CodeGen::CodeGenTypes::ConvertTypeForMem(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#34 0x00007fd2a32b1ed5 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#35 0x00007fd2a32b236f in clang::CodeGen::CodeGenTypes::ConvertTypeForMem(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#36 0x00007fd2a322b400 in (anonymous namespace)::CGRecordLowering::lower(bool) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#37 0x00007fd2a322dad7 in clang::CodeGen::CodeGenTypes::ComputeRecordLayout(clang::RecordDecl const*, llvm::StructType*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#38 0x00007fd2a32afa1d in clang::CodeGen::CodeGenTypes::ConvertRecordDeclType(clang::RecordDecl const*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#39 0x00007fd2a32b1b48 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#40 0x00007fd2a32b236f in clang::CodeGen::CodeGenTypes::ConvertTypeForMem(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#41 0x00007fd2a32b1ed5 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#42 0x00007fd2a32b236f in clang::CodeGen::CodeGenTypes::ConvertTypeForMem(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
---Type  to continue, or q  to quit--- 
#43 0x00007fd2a322b400 in (anonymous namespace)::CGRecordLowering::lower(bool) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#44 0x00007fd2a322dad7 in clang::CodeGen::CodeGenTypes::ComputeRecordLayout(clang::RecordDecl const*, llvm::StructType*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#45 0x00007fd2a32afa1d in clang::CodeGen::CodeGenTypes::ConvertRecordDeclType(clang::RecordDecl const*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#46 0x00007fd2a32b1b48 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#47 0x00007fd2a32b236f in clang::CodeGen::CodeGenTypes::ConvertTypeForMem(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#48 0x00007fd2a32b1ed5 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#49 0x00007fd2a32b236f in clang::CodeGen::CodeGenTypes::ConvertTypeForMem(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#50 0x00007fd2a322b400 in (anonymous namespace)::CGRecordLowering::lower(bool) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#51 0x00007fd2a322dad7 in clang::CodeGen::CodeGenTypes::ComputeRecordLayout(clang::RecordDecl const*, llvm::StructType*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#52 0x00007fd2a32afa1d in clang::CodeGen::CodeGenTypes::ConvertRecordDeclType(clang::RecordDecl const*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#53 0x00007fd2a32b1b48 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#54 0x00007fd2a32b236f in clang::CodeGen::CodeGenTypes::ConvertTypeForMem(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#55 0x00007fd2a32b1ed5 in clang::CodeGen::CodeGenTypes::ConvertType(clang::QualType) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#56 0x00007fd2a33002c1 in (anonymous namespace)::X86_64ABIInfo::classifyArgumentType(clang::QualType, unsigned int, unsigned int&, unsigned int&, bool) const () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#57 0x00007fd2a33016dc in (anonymous namespace)::X86_64ABIInfo::computeInfo(clang::CodeGen::CGFunctionInfo&) const () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#58 0x00007fd2a3325ab1 in clang::CodeGen::CodeGenTypes::arrangeLLVMFunctionInfo(clang::CanQual, bool, bool, llvm::ArrayRef >, clang::FunctionType::ExtInfo, llvm::ArrayRef, clang::CodeGen::RequiredArgs) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#59 0x00007fd2a33270af in arrangeLLVMFunctionInfo(clang::CodeGen::CodeGenTypes&, bool, llvm::SmallVectorImpl >&, clang::CanQual, clang::FunctionDecl const*) ()
   from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#60 0x00007fd2a3327246 in clang::CodeGen::CodeGenTypes::arrangeCXXMethodType(clang::CXXRecordDecl const*, clang::FunctionProtoType const*, clang::CXXMethodDecl const*) ()
   from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#61 0x00007fd2a328a0a7 in clang::CodeGen::CodeGenModule::EmitGlobalFunctionDefinition(clang::GlobalDecl, llvm::GlobalValue*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#62 0x00007fd2a3296dec in clang::CodeGen::CodeGenModule::EmitGlobalDefinition(clang::GlobalDecl, llvm::GlobalValue*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#63 0x00007fd2a3297ab6 in clang::CodeGen::CodeGenModule::EmitGlobal(clang::GlobalDecl) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#64 0x00007fd2a32986d9 in clang::CodeGen::CodeGenModule::EmitTopLevelDecl(clang::Decl*) [clone .part.5479] () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#65 0x00007fd2a31fee2b in clang::CodeGeneratorImpl::HandleTopLevelDecl(clang::DeclGroupRef) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#66 0x00007fd2a31d1f74 in cling::DeclCollector::HandleInterestingDecl(clang::DeclGroupRef) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#67 0x00007fd2a35efe03 in clang::ASTReader::PassInterestingDeclToConsumer(clang::Decl*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#68 0x00007fd2a363ec3c in clang::ASTReader::PassInterestingDeclsToConsumer() () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#69 0x00007fd2a35ed5d5 in clang::ASTReader::StartTranslationUnit(clang::ASTConsumer*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#70 0x00007fd2a31a76f0 in cling::IncrementalParser::Initialize(llvm::SmallVectorImpl, llvm::PointerIntPairInfo > > >&, bool) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#71 0x00007fd2a3149f52 in cling::Interpreter::Interpreter(int, char const* const*, char const*, bool, cling::Interpreter const*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#72 0x00007fd2a30fd56f in TCling::TCling(char const*, char const*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#73 0x00007fd2a30fe1c0 in CreateInterpreter () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCling.so
#74 0x00007fd83f6f57d0 in TROOT::InitInterpreter() () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCore.so
#75 0x00007fd83f6f5a3d in ROOT::Internal::GetROOT2() () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCore.so
#76 0x00007fd83f75966f in TEnv::Getvalue(char const*) const () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCore.so
#77 0x00007fd83f759a03 in TEnv::GetValue(char const*, char const*) const () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCore.so
#78 0x00007fd83f75a900 in DefaultErrorHandler(int, bool, char const*, char const*) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCore.so
#79 0x00007fd83f75a47c in ErrorHandler () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCore.so
#80 0x00007fd83f75a654 in Break(char const*, char const*, ...) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCore.so
#81 0x00007fd83f7fc065 in TUnixSystem::DispatchSignals(ESignals) () from /mu2e/ups/root/v6_12_04e/Linux64bit+3.10-2.17-e15-prof/lib/libCore.so
#82 
---Type  to continue, or q  to quit---
#83 0x00007fd83ec90c74 in artdaq::Fragment::fragmentHeader (this=0x0) at /home/ron/work/artdaqPrj/demo25-kurt2/srcs/artdaq_core/artdaq-core/Data/Fragment.hh:1160
#84 0x00007fd83e8e62ab in artdaq::Fragment::dataSize (this=0x0) at /home/ron/work/artdaqPrj/demo25-kurt2/srcs/artdaq_core/artdaq-core/Data/Fragment.hh:896
#85 artdaq::Fragment::dataSizeBytes (this=) at /home/ron/work/artdaqPrj/demo25-kurt2/srcs/artdaq_core/artdaq-core/Data/Fragment.hh:357
#86 artdaq::SharedMemoryEventManager::WriteFragmentHeader (this=0x7fd52c0586e0, frag=..., dropIfNoBuffersAvailable=dropIfNoBuffersAvailable@entry=0x0)
    at /home/ron/work/artdaqPrj/demo25-kurt2/srcs/artdaq/artdaq/DAQrate/SharedMemoryEventManager.cc:181
#87 0x00007fd83e8676b4 in artdaq::DataReceiverManager::runReceiver_ (this=0x7fd52c05aab0, source_rank=) at /home/ron/work/artdaqPrj/demo25-kurt2/srcs/artdaq/artdaq/DAQrate/DataReceiverManager.cc:238
#88 0x00007fd83d3c291d in boost::(anonymous namespace)::thread_proxy (param=) at libs/thread/src/pthread/thread.cpp:171
#89 0x00007fd83d5dfe25 in start_thread () from /lib64/libpthread.so.0
#90 0x00007fd83b2c534d in clone () from /lib64/libc.so.6
(gdb)
eflumerf commented 2 years ago

Comment by @ron003 on 2018-08-20 17:54:40

The last part of the log file is:

%MSG-i EventBuilder2_TCPSocketTransfer:  Early 20-Aug-2018 12:38:35 CDT  pre-events TCPSocket_transfer.cc:1040
listen_: New fd is 54 for source rank 13
%MSG
%MSG-i EventBuilder2_TCPSocketTransfer:  Early 20-Aug-2018 12:38:35 CDT  pre-events TCPSocket_transfer.cc:1040
[RATE LIMIT] listen_: New fd is 55 for source rank 12
%MSG
%MSG-i EventBuilder2:  Early 20-Aug-2018 12:38:50 CDT pre-events report_metric.cc:158
Periodic report: 15 active metrics:
        EventBuilder2.Average Event Size: 159138560.000000 Bytes,
        EventBuilder2.Event Rate: 0.000000 Events/s,
        EventBuilder2.Events Released to art this run: 0 Events,
        EventBuilder2.Events Released to art this subrun: 0 Events,
        EventBuilder2.Incomplete Events Released to art this run: 0 Events,
        EventBuilder2.Incomplete Events Released to art this subrun: 0 Events,
        EventBuilder2.Input Fragment Rate: 0.000001 Fragments/s,
        EventBuilder2.Run Number: 305.000100 Run:Subrun,
        EventBuilder2.Shared Memory Available %: 100.000000 %,
        EventBuilder2.Shared Memory Available Buffers: 3 buffers,
        EventBuilder2.Shared Memory Full %: 0.000000 %,
        EventBuilder2.Shared Memory Full Buffers: 0 buffers,
        EventBuilder2.Shared Memory Pending Buffers: 0 buffers,
        EventBuilder2.Shared Memory Reading Buffers: 0 buffers,
        EventBuilder2.Tokens sent: 5 Tokens
%MSG
%MSG-w EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:1060
Active event 451 is stale. Scheduling release of incomplete event (missing 3 Fragments) to art.
%MSG
%MSG-w EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:1060
Active event 449 is stale. Scheduling release of incomplete event (missing 2 Fragments) to art.
%MSG
%MSG-e EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:907
Event 451 has already been marked "Incomplete" and sent to art!
%MSG
%MSG-e EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:173
Dropping fragment with sequence id 451 and fragment id 4 because data taking has already passed this event.
%MSG
%MSG-e EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:907
Event 449 has already been marked "Incomplete" and sent to art!
%MSG
%MSG-e EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:173
Dropping fragment with sequence id 449 and fragment id 8 because data taking has already passed this event.
%MSG
%MSG-e EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:907
Event 449 has already been marked "Incomplete" and sent to art!
%MSG
%MSG-e EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:173
Dropping fragment with sequence id 449 and fragment id 10 because data taking has already passed this event.
%MSG
%MSG-i EventBuilder2:  Early 20-Aug-2018 12:39:10 CDT pre-events report_metric.cc:158
Periodic report: 14 active metrics:
        EventBuilder2.Average Event Size: 159138560.000000 Bytes,
        EventBuilder2.Event Rate: 1.571551 Events/s,
        EventBuilder2.Events Released to art this run: 24 Events,
        EventBuilder2.Events Released to art this subrun: 24 Events,
        EventBuilder2.Incomplete Events Released to art this run: 0 Events,
        EventBuilder2.Incomplete Events Released to art this subrun: 0 Events,
        EventBuilder2.Input Fragment Rate: 34.641673 Fragments/s,
        EventBuilder2.Run Number: 305.000100 Run:Subrun,
        EventBuilder2.Shared Memory Available %: 60.000000 %,
        EventBuilder2.Shared Memory Available Buffers: NOT REPORTED,
        EventBuilder2.Shared Memory Full %: 0.000000 %,
        EventBuilder2.Shared Memory Full Buffers: 0 buffers,
        EventBuilder2.Shared Memory Pending Buffers: 2 buffers,
        EventBuilder2.Shared Memory Reading Buffers: 0 buffers,
        EventBuilder2.Tokens sent: 27 Tokens
%MSG
%MSG-e EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:907
Event 451 has already been marked "Incomplete" and sent to art!
%MSG
%MSG-e EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:907
Event 449 has already been marked "Incomplete" and sent to art!
%MSG
%MSG-e EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:907
Event 451 has already been marked "Incomplete" and sent to art!
%MSG
%MSG-e EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:173
Dropping fragment with sequence id 451 and fragment id 10 because data taking has already passed this event.
%MSG
%MSG-e EventBuilder2_SharedMemoryEventManager:  Early 20-Aug-2018 12:39:10 CDT  pre-events SharedMemoryEventManager.cc:907
Event 451 has already been marked "Incomplete" and sent to art!
%MSG