crash in monitoring hists after recent pull requests

[aaust]

I am observing almost 50% of the jobs crashing with the current master:

*** Error in `hd_root': double free or corruption (fasttop): 0x00002b2760106b50 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7cfe1)[0x2b2707edffe1]
hd_root(_ZNSt6vectorImSaImEE19_M_emplace_back_auxIImEEEvDpOT_+0x83)[0xf884a3]
hd_root(_ZN16DEventSourceREST23Extract_DTrackTimeBasedEPN6hddm_r4HDDMEPN4jana8JFactoryI15DTrackTimeBasedEEPNS3_10JEventLoopE+0x1481)[0xf87251]
hd_root(_ZN16DEventSourceREST10GetObjectsERN4jana6JEventEPNS0_13JFactory_baseE+0xcbf)[0xf8807f]
hd_root(_ZN4jana6JEvent10GetObjectsI15DTrackTimeBasedEE8jerror_tRSt6vectorIPKT_SaIS7_EEPNS_13JFactory_baseE+0x2c)[0x697c1c]
hd_root(_ZN4jana10JEventLoop14GetFromFactoryI15DTrackTimeBasedEEPNS_8JFactoryIT_EERSt6vectorIPKS4_SaIS9_EEPKcRNS0_13data_source_tEb+0x102)[0x697ee2]
hd_root(_ZN4jana10JEventLoop3GetI15DTrackTimeBasedEEPNS_8JFactoryIT_EERSt6vectorIPKS4_SaIS9_EEPKcb+0x19c)[0x6981fc]
hd_root(_ZN40DHistogramAction_NumReconstructedObjects14Perform_ActionEPN4jana10JEventLoopEPKN9DAnalysis14DParticleComboE+0x8a)[0x74c6da]
/work/halld2/home/gxproj6/builds/halld_recon/monitoring_launch//Linux_CentOS7-x86_64-gcc4.8.5/plugins/monitoring_hists.so(_ZN32DEventProcessor_monitoring_hists4evntEPN4jana10JEventLoopEm+0x24b)[0x2b271e12656b]
hd_root(_ZN4jana10JEventLoop8OneEventEv+0x34a)[0xfe528a]
hd_root(_ZN4jana10JEventLoop4LoopEv+0x64)[0xfe6224]
hd_root(_Z12LaunchThreadPv+0x5a)[0xfbd4da]
/lib64/libpthread.so.0(+0x7dc5)[0x2b270722adc5]
/lib64/libc.so.6(clone+0x6d)[0x2b2707f5921d]

Here are the first few lines demangled:

void std::vector<unsigned long, std::allocator<unsigned long> >::_M_emplace_back_aux<unsigned long>(unsigned long&&)
DEventSourceREST::Extract_DTrackTimeBased(hddm_r::HDDM*, jana::JFactory<DTrackTimeBased>*, jana::JEventLoop*)
DEventSourceREST::GetObjects(jana::JEvent&, jana::JFactory_base*)
jerror_t jana::JEvent::GetObjects<DTrackTimeBased>(std::vector<DTrackTimeBased const*, std::allocator<DTrackTimeBased const*> >&, jana::JFactory_base*)

Maybe related to pull request #121 ?

[markito3]

That pull request was merged on Feb. 26 at 5 pm. Commit 13c806c5c . Can you try the nightly build from Feb. 26 and see if the problem goes away?

[markito3]

FYI, you can find information on the nightly build here.

[aaust]

Thanks Mark. I do not see any crashes with the nightly build from 2019-02-26.

@staylorjlab suggested a fix in the branch TrackExtractorFix, but it still crashes with it.

[aaust]

No more crashes after the revert.