Execution of "enable-mapfile" option results in crashes

paulmking commented 5 years ago

Environment: (where does this bug occur, have you tried other environments)

branch (master for latest released): develop (see below for discussion)
revision (HEAD for most recent): HEAD (commit 0cce610)
OS or system: ifarm (and adaq3, but description below is on ifarm)
Special ROOT or Geant4 versions? Using "JLab software" 2.3 on ifarm

Steps to reproduce: (give a step by step account of how to trigger the bug)

With the default configuration options, the execution crashes because TMapFile requires the ROOT "New" library which is not enabled by default.
1. Generate a test data file using the mock data generator: _./build/qwmockdatagenerator -r 10 -e 1:10000 --config qwparity_simple.conf --detectors mockdetectors.map
2. Trying to analyze that file crashes: _./build/qwparity --enable-mapfile -r 10 --config qwparity_simple.conf --detectors mockdetectors.map The result is: Error in : no memory mapped file capability available Use rootn.exe or link application against "-lNew" ================== RealTime Producer Memory Map File ================= Memory mapped file: /dev/shm//QwMemMapFile.map Title: RealTime Producer File Option: file closed
  
  QwRootFile::ConstructHistograms::detectors address 0x7ffcfc5e7f20 and its name mps_histo
Break segmentation violation
In the branch 'bugfix-using-TMapFile', I have modified the cmake to include the ROOT "New" library. Historically, the way to use the "New" library was to have it occur first in the linking; at the starting point of the branch, it is not liked early, it is at the end of the ROOT library linking process.
1. Trying the same analysis results in immediate crash (although even qwparity --help crashes): pking@ifarm1401.jlab.org> ./build_new/qwparity --enable-mapfile -r 10 --config qwparity_simple.conf --detectors mock_detectors.map
Break segmentation violation

wdconinc commented 5 years ago

Not sure if just adding it to find_package will be sufficient. We might need to modify FindROOT.cmake (https://github.com/JeffersonLab/japan/blob/ce06f358537c86fc82aa54d2343b56857267386f/cmake/modules/FindROOT.cmake#L103) to call root-config --new --libs too.

wdconinc commented 5 years ago

Note that root-config --new --libs returns -lNew, but root-config --libs --new does not.

paulmking commented 5 years ago

I had tried changing the FindROOT in two different ways:

When building the list of libraries to export, insert "New" at the beginning of the list if it is present in the requested libraries, still requesting "New" as a required library in the main cmake That didn't work any better than the default FindROOT.
Changing FindROOT to use root-config --new --libs to create the library list, and revert the main cmake. This still results in ./build_newer/qwparity --help segfaulting. The libNew is near the top of ldd ./build_newer/qwparity, but is below the boost libraries, mysqlpp (if present), and the japan library itself:

pking@ifarm1401.jlab.org> ldd ./build_newer/qwparity linux-vdso.so.1 => (0x00007ffc381db000) libQwAnalysis.so => /home/pking/PREX/japan/build_newer/libQwAnalysis.so (0x00007f4482220000) libmysqlpp.so.3 => /group/qweak/QwAnalysis/Linux_CentOS6.5-x86_64/MySQL++/local/lib/libmysqlpp.so.3 (0x00007f4481fdb000) libboost_program_options-mt.so.1.53.0 => /lib64/libboost_program_options-mt.so.1.53.0 (0x00007f4481d51000) libboost_filesystem-mt.so.1.53.0 => /lib64/libboost_filesystem-mt.so.1.53.0 (0x00007f4481b39000) libboost_system-mt.so.1.53.0 => /lib64/libboost_system-mt.so.1.53.0 (0x00007f4481935000) libboost_regex-mt.so.1.53.0 => /lib64/libboost_regex-mt.so.1.53.0 (0x00007f4481632000) libNew.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libNew.so.6.14 (0x00007f448142c000) libCore.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libCore.so.6.14 (0x00007f4480d9d000) libImt.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libImt.so.6.14 (0x00007f4480b8f000) libRIO.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libRIO.so.6.14 (0x00007f44805f9000) libNet.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libNet.so.6.14 (0x00007f4480329000) libHist.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libHist.so.6.14 (0x00007f447fd92000) libGraf.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libGraf.so.6.14 (0x00007f447fa42000) libGraf3d.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libGraf3d.so.6.14 (0x00007f447f79b000) libGpad.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libGpad.so.6.14 (0x00007f447f4e0000) libROOTDataFrame.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libROOTDataFrame.so.6.14 (0x00007f447f27d000) libROOTVecOps.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libROOTVecOps.so.6.14 (0x00007f447efaa000) libTree.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libTree.so.6.14 (0x00007f447ec4e000) libTreePlayer.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libTreePlayer.so.6.14 (0x00007f447e901000) libRint.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libRint.so.6.14 (0x00007f447e6d9000) libPostscript.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libPostscript.so.6.14 (0x00007f447e46c000) libMatrix.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libMatrix.so.6.14 (0x00007f447e102000) libPhysics.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libPhysics.so.6.14 (0x00007f447deb3000) libMathCore.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libMathCore.so.6.14 (0x00007f447dab0000) libThread.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libThread.so.6.14 (0x00007f447d85e000) libMultiProc.so.6.14 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libMultiProc.so.6.14 (0x00007f447d64f000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f447d347000) libm.so.6 => /lib64/libm.so.6 (0x00007f447d045000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f447ce2e000) libc.so.6 => /lib64/libc.so.6 (0x00007f447ca6d000) libevio.so => /home/pking/PREX/japan/build_newer/evio/libevio.so (0x00007f447c85f000) libmysqlclient.so.16 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/mysql/lib/libmysqlclient.so.16 (0x00007f447c4d8000) librt.so.1 => /lib64/librt.so.1 (0x00007f447c2d0000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f447c0b3000) libicuuc.so.50 => /lib64/libicuuc.so.50 (0x00007f447bd3a000) libicui18n.so.50 => /lib64/libicui18n.so.50 (0x00007f447b93b000) libicudata.so.50 => /lib64/libicudata.so.50 (0x00007f447a366000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f447a105000) libpcreposix.so.0 => /lib64/libpcreposix.so.0 (0x00007f4479f02000) libz.so.1 => /lib64/libz.so.1 (0x00007f4479ceb000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f4479ae7000) /lib64/ld-linux-x86-64.so.2 (0x00007f44826db000) libtbb.so.2 => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libtbb.so.2 (0x00007f44798aa000) libssl.so.10 => /lib64/libssl.so.10 (0x00007f447963c000) libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f4479255000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f447901e000) libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007f4478d77000) libvdt.so => /site/12gev_phys/2.3/Linux_CentOS7.2.1511-x86_64-gcc4.8.5/root/6.14.04/lib/libvdt.so (0x00007f4478b70000) libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f4478955000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f4478709000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f4478423000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f447821f000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f4477fed000) libfreebl3.so => /lib64/libfreebl3.so (0x00007f4477de9000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f4477bda000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f44779d6000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f44777bb000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f4477596000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f4477370000)

wdconinc commented 5 years ago

Putting libNew.so first does not fix the problem. Originally libNew.so was very last in libQwAnalysis.so link and just before rpath in qwparity. Putting it all the way at the start of the so list still results in seg fault. Output log and bt below the commands.

13:45:23 wdconinc@herakles ~/git/japan/build $ /usr/bin/c++ -fPIC -O2 -g -DNDEBUG -fno-omit-frame-pointer  -shared -Wl,-soname,libQwAnalysis.so -o libQwAnalysis.so CMakeFiles/QwAnalysis.dir/Analysis/src/MQwCodaControlEvent.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/MQwF1TDC.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/MQwMockable.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/MQwV775TDC.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwColor.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwDBInterface.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwDatabase.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwEPICSEvent.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwEventBuffer.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwHistogramHelper.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwLog.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwOmnivore.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwOptions.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwPMT_Channel.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwParameterFile.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwPromptSummary.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwRint.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwRootFile.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwRunCondition.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwSIS3320_Accumulator.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwSIS3320_LogicalAccumulator.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwScaler_Channel.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwSubsystemArray.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwTypes.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/QwVQWK_Channel.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/VQwHardwareChannel.cc.o CMakeFiles/QwAnalysis.dir/Analysis/src/VQwSubsystem.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/LRBCorrector.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/LinReg_Bevington_Pebay.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwBCM.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwBPMCavity.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwBPMStripline.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwBeamDetectorID.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwBeamLine.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwBlindDetectorArray.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwBlinder.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwClock.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwCombinedBCM.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwCombinedBPM.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwCombinedPMT.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwCombiner.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwCombinerSubsystem.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwComptonElectronDetector.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwCorrelator.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwDetectorArray.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwEnergyCalculator.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwEventRing.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwFakeHelicity.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwHaloMonitor.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwHelicity.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwHelicityPattern.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwIntegrationPMT.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwLinearDiodeArray.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwMollerDetector.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwParityDB.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwQPD.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwScaler.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwSubsystemArrayParity.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/QwkRegBlueCorrelator.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/VQwBCM.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/VQwBPM.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/VQwClock.cc.o CMakeFiles/QwAnalysis.dir/Parity/src/VQwDataHandler.cc.o CMakeFiles/QwAnalysis.dir/gitinfo.cc.o CMakeFiles/QwAnalysis.dir/QwPromptSummaryDict.cxx.o -Wl,-rpath,/home/wdconinc/git/japan/build/evio:/usr/local/root/pro/lib: evio/libevio.so /usr/lib/libmysqlpp.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_filesystem.so /usr/lib/x86_64-linux-gnu/libboost_system.so /usr/lib/x86_64-linux-gnu/libboost_regex.so /usr/local/root/pro/lib/libNew.so /usr/local/root/pro/lib/libCore.so /usr/local/root/pro/lib/libImt.so /usr/local/root/pro/lib/libRIO.so /usr/local/root/pro/lib/libNet.so /usr/local/root/pro/lib/libHist.so /usr/local/root/pro/lib/libGraf.so /usr/local/root/pro/lib/libGraf3d.so /usr/local/root/pro/lib/libGpad.so /usr/local/root/pro/lib/libROOTDataFrame.so /usr/local/root/pro/lib/libROOTVecOps.so /usr/local/root/pro/lib/libTree.so /usr/local/root/pro/lib/libTreePlayer.so /usr/local/root/pro/lib/libRint.so /usr/local/root/pro/lib/libPostscript.so /usr/local/root/pro/lib/libMatrix.so /usr/local/root/pro/lib/libPhysics.so /usr/local/root/pro/lib/libMathCore.so /usr/local/root/pro/lib/libThread.so /usr/local/root/pro/lib/libMultiProc.so
13:46:18 wdconinc@herakles ~/git/japan/build $ /usr/bin/c++  -O2 -g -DNDEBUG -fno-omit-frame-pointer   CMakeFiles/qwparity.dir/Parity/main/QwParity.cc.o  -o qwparity -Wl,-rpath,/home/wdconinc/git/japan/build:/home/wdconinc/git/japan/build/evio:/usr/local/root/pro/lib: libQwAnalysis.so /usr/lib/libmysqlpp.so /usr/lib/x86_64-linux-gnu/libboost_program_options.so /usr/lib/x86_64-linux-gnu/libboost_filesystem.so /usr/lib/x86_64-linux-gnu/libboost_system.so /usr/lib/x86_64-linux-gnu/libboost_regex.so /usr/local/root/pro/lib/libNew.so /usr/local/root/pro/lib/libCore.so /usr/local/root/pro/lib/libImt.so /usr/local/root/pro/lib/libRIO.so /usr/local/root/pro/lib/libNet.so /usr/local/root/pro/lib/libHist.so /usr/local/root/pro/lib/libGraf.so /usr/local/root/pro/lib/libGraf3d.so /usr/local/root/pro/lib/libGpad.so /usr/local/root/pro/lib/libROOTDataFrame.so /usr/local/root/pro/lib/libROOTVecOps.so /usr/local/root/pro/lib/libTree.so /usr/local/root/pro/lib/libTreePlayer.so /usr/local/root/pro/lib/libRint.so /usr/local/root/pro/lib/libPostscript.so /usr/local/root/pro/lib/libMatrix.so /usr/local/root/pro/lib/libPhysics.so /usr/local/root/pro/lib/libMathCore.so /usr/local/root/pro/lib/libThread.so /usr/local/root/pro/lib/libMultiProc.so -Wl,-rpath-link,/home/wdconinc/git/japan/build/evio

Log

13:43:29 wdconinc@herakles ~/git/japan (bugfix-using-TMapFile *$=) $ build/qwparity --enable-mapfile -r 10 --config qwparity_simple.conf --detectors mock_detectors.map
/home/wdconinc/qweak/QwScratch/setupfiles is a directory; adding it to the search path
/home/wdconinc/git/japan/Parity/prminput is a directory; adding it to the search path
/home/wdconinc/git/japan/Analysis/prminput is a directory; adding it to the search path
Adding user-defined configuration file qwparity.conf
Adding user-defined configuration file qweak_mysql.conf
Config files:
qwparity.conf
qweak_mysql.conf
Warning: Overriding the default configuration files with user-defined configuration file qwparity_simple.conf
Parameter file: /home/wdconinc/git/japan/Parity/prminput/qwparity_simple.conf
tree-trim is disabled
histo-trim is disabled 
Parameter file: /home/wdconinc/git/japan/Parity/prminput/parity_hist.in
Opening data file:  /home/wdconinc/qweak/QwScratch/data/QwMock_10.log
Warning: Overriding the default configuration files with user-defined configuration file qwparity_simple.conf
Parameter file: /home/wdconinc/git/japan/Parity/prminput/qwparity_simple.conf
Parameter file: /home/wdconinc/git/japan/Parity/prminput/mock_detectors.map
Loading subsystems from mock_detectors.map.
Creating subsystem of type QwHelicity with name Helicity Info.
Parameter file: /home/wdconinc/git/japan/Parity/prminput/mock_qweak_helicity.map
 **** Input Register Mode **** 
Creating subsystem of type QwBeamLine with name Injector BeamLine.
Parameter file: /home/wdconinc/git/japan/Parity/prminput/mock_qweak_beamline.map
Error: At end of processing the combined device 
Error: At end of processing the combined device 
Error: At end of processing the combined device 
Variables to publish:
q_targ combobcm bcm_target c
x_targ combobpm bpm_target x
y_targ combobpm bpm_target y
xp_targ combobpm bpm_target xp
yp_targ combobpm bpm_target yp
e_targ comboenergy target_energy e
Parameter file: /home/wdconinc/git/japan/Parity/prminput/mock_qweak_pedestal.map
Parameter file: /home/wdconinc/git/japan/Parity/prminput/mock_beamline_geometry.map
Parameter file: /home/wdconinc/git/japan/Parity/prminput/mock_beamline_eventcuts.map
Creating subsystem of type QwDetectorArray with name Main Detector.
Parameter file: /home/wdconinc/git/japan/Parity/prminput/mock_moller_maindet_adc.map
Variables to publish:
Parameter file: /home/wdconinc/git/japan/Parity/prminput/mock_moller_maindet_pedestal.map
Parameter file: /home/wdconinc/git/japan/Parity/prminput/mock_moller_maindet_eventcuts.map
QwHelicity::ProcessOptions
 fPatternBits 0x69
List of published values:
e_targ: published-value
q_targ: published-value
x_targ: published-value
xp_targ: published-value
y_targ: published-value
yp_targ: published-value
Blinding parameters have been calculated.
QwBlinder::InitTestValues(): A total of 10 test values have been calculated successfully.
Warning: enable-lrbcorrection is set to false.  Skipping LoadChannelMap for LRBCorrector
QwHelicity::MaxPatternPhase = 4
QwBlinder:  First set target blindability to NotBlindable
================== RealTime Producer Memory Map File =================
Memory mapped file:   /dev/shm//QwMemMapFile.map
Title:                RealTime Producer File
Option:               CREATE
Mapped Memory region: 0x7fe3f215d000 - 0x7fe40215d000 (256.00 MB)
Current breakval:     0x7fe3f2168000
======================================================================
QwRootFile::ConstructHistograms::detectors address 0x7fff04ce94d0 and its name mps_histo
QwRootFile::ConstructHistograms::detectors address 0x7fff04ce9ce0 and its name hel_histo
New tree: Mps_Tree, MPS event data tree
New tree: Hel_Tree, Helicity event data tree
New tree: Hel_Tree_Reg, Helicity event data tree (regressed)
Error in <TTree::SetBasketSize>: unknown branch -> '*'
New tree: Hel_Tree_LRB, Helicity event data tree (regressed)
Error in <TTree::SetBasketSize>: unknown branch -> '*'
Finding first EPICS event
Processing event 10000 (0.066 ms per event with 93.5084% efficiency)
Normal end of file /home/wdconinc/qweak/QwScratch/data/QwMock_10.log encountered
The EPICS variable label and type arrays are not the same size!
EPICS readback may be corrupted!
Rewinding stream
Opening data file:  /home/wdconinc/qweak/QwScratch/data/QwMock_10.log
Error: QwHelicity::ProcessEvent read event# (1) is not  old_event#+1; missed 1 gates
Warning: QwHelicity::IsGoodPatternNumber:  This is not a good pattern number. New = -1 Old = -1
Warning: QwHelicity::ResetPredictor:  Resetting helicity prediction!
Warning: QwHelicity::CollectRandBits30:  While collecting the seed, we encountered non continuous events: Need to reset the seed collecting 
Warning: QwHelicityPattern::LoadEventData:  Reduced event phase number is less than zero; ignore this event.
Warning: QwHelicity::IsGoodPatternNumber:  This is not a good pattern number. New = -1 Old = -1
Warning: QwHelicity::ResetPredictor:  Resetting helicity prediction!
Warning: QwHelicity::CollectRandBits30:  While collecting the seed, we encountered non continuous events: Need to reset the seed collecting 
Warning: QwHelicityPattern::LoadEventData:  Reduced event phase number is less than zero; ignore this event.
Warning: QwHelicity::IsGoodPatternNumber:  This is not a good pattern number. New = -1 Old = -1
Warning: QwHelicity::ResetPredictor:  Resetting helicity prediction!
Warning: QwHelicity::CollectRandBits30:  While collecting the seed, we encountered non continuous events: Need to reset the seed collecting 
Warning: QwHelicityPattern::LoadEventData:  Reduced event phase number is less than zero; ignore this event.
Collecting information (-) from event #4 to generate helicity seed (need 30 bit, so far got 0 bits )
Warning: QwHelicity::IsGoodPatternNumber:  This is not a good pattern number. New = 1 Old = -1
Warning: QwHelicity::ResetPredictor:  Resetting helicity prediction!
Warning: QwHelicity::CollectRandBits30:  While collecting the seed, we encountered non continuous events: Need to reset the seed collecting 
Collecting information (-) from event #8 to generate helicity seed (need 30 bit, so far got 0 bits )
Collecting information (+) from event #12 to generate helicity seed (need 30 bit, so far got 1 bits )
Collecting information (+) from event #16 to generate helicity seed (need 30 bit, so far got 2 bits )
Collecting information (-) from event #20 to generate helicity seed (need 30 bit, so far got 3 bits )
Collecting information (-) from event #24 to generate helicity seed (need 30 bit, so far got 4 bits )
Collecting information (+) from event #28 to generate helicity seed (need 30 bit, so far got 5 bits )
Collecting information (+) from event #32 to generate helicity seed (need 30 bit, so far got 6 bits )
Collecting information (-) from event #36 to generate helicity seed (need 30 bit, so far got 7 bits )
Collecting information (-) from event #40 to generate helicity seed (need 30 bit, so far got 8 bits )
Collecting information (+) from event #44 to generate helicity seed (need 30 bit, so far got 9 bits )
Collecting information (+) from event #48 to generate helicity seed (need 30 bit, so far got 10 bits )
Collecting information (+) from event #52 to generate helicity seed (need 30 bit, so far got 11 bits )
Collecting information (-) from event #56 to generate helicity seed (need 30 bit, so far got 12 bits )
Collecting information (-) from event #60 to generate helicity seed (need 30 bit, so far got 13 bits )
Collecting information (+) from event #64 to generate helicity seed (need 30 bit, so far got 14 bits )
Collecting information (+) from event #68 to generate helicity seed (need 30 bit, so far got 15 bits )
Collecting information (+) from event #72 to generate helicity seed (need 30 bit, so far got 16 bits )
Collecting information (+) from event #76 to generate helicity seed (need 30 bit, so far got 17 bits )
Collecting information (+) from event #80 to generate helicity seed (need 30 bit, so far got 18 bits )
Collecting information (-) from event #84 to generate helicity seed (need 30 bit, so far got 19 bits )
Collecting information (-) from event #88 to generate helicity seed (need 30 bit, so far got 20 bits )
Collecting information (+) from event #92 to generate helicity seed (need 30 bit, so far got 21 bits )
Collecting information (-) from event #96 to generate helicity seed (need 30 bit, so far got 22 bits )
Collecting information (-) from event #100 to generate helicity seed (need 30 bit, so far got 23 bits )
Collecting information (-) from event #104 to generate helicity seed (need 30 bit, so far got 24 bits )
Collecting information (-) from event #108 to generate helicity seed (need 30 bit, so far got 25 bits )
Collecting information (+) from event #112 to generate helicity seed (need 30 bit, so far got 26 bits )
Collecting information (+) from event #116 to generate helicity seed (need 30 bit, so far got 27 bits )
Collecting information (+) from event #120 to generate helicity seed (need 30 bit, so far got 28 bits )
Collecting information (+) from event #124 to generate helicity seed (need 30 bit, so far got 29 bits )

 *** Break *** segmentation violation
sh: 1: ���: not found

===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007fe4106803b7 in __GI___waitpid (pid=4037, stat_loc=stat_loc
entry=0x7fff04ce5f98, options=options
entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1  0x00007fe4105f1f17 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:149
#2  0x00007fe4111f6980 in TUnixSystem::Exec (shellcmd=<optimized out>, this=0x559c7e139808) at /usr/local/src/root/root-6.16.00/core/unix/src/TUnixSystem.cxx:2119
#3  TUnixSystem::StackTrace (this=0x559c7e139808) at /usr/local/src/root/root-6.16.00/core/unix/src/TUnixSystem.cxx:2413
#4  0x00007fe4111f93c4 in TUnixSystem::DispatchSignals (this=0x559c7e139808, sig=kSigSegmentationViolation) at /usr/local/src/root/root-6.16.00/core/unix/src/TUnixSystem.cxx:3644
#5  <signal handler called>
#6  0x0000000000000000 in ?? ()
#7  0x00007fe4111995a5 in TClass::GetBaseClassOffset (isDerivedObject=true, address=0x0, toBase=0x559c7e840028, this=0x559c85e7b678) at /usr/local/src/root/root-6.16.00-build/include/TClass.h:402
#8  TClass::GetBaseClassOffset (this=0x559c85e7b678, toBase=0x559c7e840028, address=0x0, isDerivedObject=<optimized out>) at /usr/local/src/root/root-6.16.00/core/meta/src/TClass.cxx:2708
#9  0x00007fe410bd9e22 in TBufferIO::WriteObjectAny (cacheReuse=true, ptrClass=0x559c7e840028, obj=0x559c89dc76f8, this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferIO.cxx:518
#10 TBufferIO::WriteObjectAny (this=this
entry=0x7fe3f2164f08, obj=obj
entry=0x559c89dc76f8, ptrClass=0x559c7e840028, cacheReuse=cacheReuse
entry=true) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferIO.cxx:492
#11 0x00007fe41116c0a0 in operator<< (obj=0x559c89dc76f8, buf=...) at /usr/local/src/root/root-6.16.00-build/include/TBuffer.h:409
#12 TObjArray::Streamer (this=0x559c85ba6410, b=...) at /usr/local/src/root/root-6.16.00/core/cont/src/TObjArray.cxx:486
#13 0x00007fe410bcc665 in TClass::Streamer (onfile_class=0x0, b=..., obj=0x559c85ba6410, this=0x559c7ed1fa28) at /usr/local/src/root/root-6.16.00-build/include/TClass.h:568
#14 TBufferFile::WriteFastArray (streamer=0x0, n=1, cl=0x559c7ed1fa28, start=<optimized out>, this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:2256
#15 TBufferFile::WriteFastArray (this=0x7fe3f2164f08, start=<optimized out>, cl=0x559c7ed1fa28, n=1, streamer=<optimized out>) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:2243
#16 0x00007fe410e19cb1 in TStreamerInfo::WriteBufferAux<char**> (this=0x7fe3fd243c08, b=..., arr=
0x7fff04ce8bc0: 0x7fff04ce8bb8, compinfo=0x7fe3fd24bfe0, first=first
entry=0, last=last
entry=1, narr=1, eoffset=0, arrayMode=0) at /usr/local/src/root/root-6.16.00-build/include/TString.h:239
#17 0x00007fe410c81130 in TStreamerInfoActions::GenericWriteAction (buf=..., addr=<optimized out>, config=<optimized out>) at /usr/local/src/root/root-6.16.00/io/io/src/TStreamerInfoActions.cxx:192
#18 0x00007fe410bd3202 in TStreamerInfoActions::TConfiguredAction::operator() (this=0x7fe3fd2389c8, this=0x7fe3fd2389c8, object=0x559c85ba62c8, buffer=...) at /usr/local/src/root/root-6.16.00-build/include/TStreamerInfoActions.h:124
#19 TBufferFile::ApplySequence (obj=0x559c85ba62c8, sequence=..., this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3564
#20 TBufferFile::ApplySequence (obj=0x559c85ba62c8, sequence=..., this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3546
#21 TBufferFile::WriteClassBuffer (this=this
entry=0x7fe3f2164f08, cl=0x559c883785e8, pointer=pointer
entry=0x559c85ba62c8) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3533
#22 0x00007fe410a32f2f in TBranch::Streamer (this=0x559c85ba62c8, b=...) at /usr/local/src/root/root-6.16.00/tree/tree/src/TBranch.cxx:2767
#23 0x00007fe410bd2022 in TClass::Streamer (onfile_class=0x0, b=..., obj=0x559c85ba62c8, this=0x559c883785e8) at /usr/local/src/root/root-6.16.00-build/include/TClass.h:568
#24 TBufferFile::WriteObjectClass (this=0x7fe3f2164f08, actualObjectStart=0x559c85ba62c8, actualClass=0x559c883785e8, cacheReuse=<optimized out>) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:2552
#25 0x00007fe410bd9e3d in TBufferIO::WriteObjectAny (cacheReuse=true, ptrClass=0x559c7e840028, obj=0x559c85ba62c8, this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferIO.cxx:519
#26 TBufferIO::WriteObjectAny (this=this
entry=0x7fe3f2164f08, obj=obj
entry=0x559c85ba62c8, ptrClass=0x559c7e840028, cacheReuse=cacheReuse
entry=true) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferIO.cxx:492
#27 0x00007fe41116c0a0 in operator<< (obj=0x559c85ba62c8, buf=...) at /usr/local/src/root/root-6.16.00-build/include/TBuffer.h:409
#28 TObjArray::Streamer (this=0x559c84222fc0, b=...) at /usr/local/src/root/root-6.16.00/core/cont/src/TObjArray.cxx:486
#29 0x00007fe410bcc665 in TClass::Streamer (onfile_class=0x0, b=..., obj=0x559c84222fc0, this=0x559c7ed1fa28) at /usr/local/src/root/root-6.16.00-build/include/TClass.h:568
#30 TBufferFile::WriteFastArray (streamer=0x0, n=1, cl=0x559c7ed1fa28, start=<optimized out>, this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:2256
#31 TBufferFile::WriteFastArray (this=0x7fe3f2164f08, start=<optimized out>, cl=0x559c7ed1fa28, n=1, streamer=<optimized out>) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:2243
#32 0x00007fe410e19cb1 in TStreamerInfo::WriteBufferAux<char**> (this=0x7fe3f2209d08, b=..., arr=
0x7fff04ce8f80: 0x7fff04ce8f78, compinfo=0x7fe3fd23e5a0, first=first
entry=0, last=last
entry=1, narr=1, eoffset=0, arrayMode=0) at /usr/local/src/root/root-6.16.00-build/include/TString.h:239
#33 0x00007fe410c81130 in TStreamerInfoActions::GenericWriteAction (buf=..., addr=<optimized out>, config=<optimized out>) at /usr/local/src/root/root-6.16.00/io/io/src/TStreamerInfoActions.cxx:192
#34 0x00007fe410bd3202 in TStreamerInfoActions::TConfiguredAction::operator() (this=0x7fe3fd239aa8, this=0x7fe3fd239aa8, object=0x559c84222e68, buffer=...) at /usr/local/src/root/root-6.16.00-build/include/TStreamerInfoActions.h:124
#35 TBufferFile::ApplySequence (obj=0x559c84222e68, sequence=..., this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3564
#36 TBufferFile::ApplySequence (obj=0x559c84222e68, sequence=..., this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3546
#37 TBufferFile::WriteClassBuffer (this=this
entry=0x7fe3f2164f08, cl=0x559c842234c8, pointer=pointer
entry=0x559c84222e68) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3533
#38 0x00007fe410aa48fc in TTree::Streamer (this=0x559c84222e68, b=...) at /usr/local/src/root/root-6.16.00/tree/tree/src/TTree.cxx:9216
#39 0x00007fe410c60087 in TMapFile::Update (this=0x7fe3f2164008, obj=obj
entry=0x0) at /usr/local/src/root/root-6.16.00/io/io/src/TMapFile.cxx:649
#40 0x0000559c7deabcca in QwRootFile::Update (this=0x559c7f9bbd48) at /home/wdconinc/git/japan/Analysis/include/QwRootFile.h:412
#41 QwRootFile::FillHistograms<QwSubsystemArrayParity> (this=this
entry=0x559c7f9bbd48, object=...) at /home/wdconinc/git/japan/Analysis/include/QwRootFile.h:333
#42 0x0000559c7dea4204 in main (argc=<optimized out>, argv=<optimized out>) at /home/wdconinc/git/japan/Parity/main/QwParity.cc:284
===========================================================

The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum http://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at http://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#6  0x0000000000000000 in ?? ()
#7  0x00007fe4111995a5 in TClass::GetBaseClassOffset (isDerivedObject=true, address=0x0, toBase=0x559c7e840028, this=0x559c85e7b678) at /usr/local/src/root/root-6.16.00-build/include/TClass.h:402
#8  TClass::GetBaseClassOffset (this=0x559c85e7b678, toBase=0x559c7e840028, address=0x0, isDerivedObject=<optimized out>) at /usr/local/src/root/root-6.16.00/core/meta/src/TClass.cxx:2708
#9  0x00007fe410bd9e22 in TBufferIO::WriteObjectAny (cacheReuse=true, ptrClass=0x559c7e840028, obj=0x559c89dc76f8, this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferIO.cxx:518
#10 TBufferIO::WriteObjectAny (this=this
entry=0x7fe3f2164f08, obj=obj
entry=0x559c89dc76f8, ptrClass=0x559c7e840028, cacheReuse=cacheReuse
entry=true) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferIO.cxx:492
#11 0x00007fe41116c0a0 in operator<< (obj=0x559c89dc76f8, buf=...) at /usr/local/src/root/root-6.16.00-build/include/TBuffer.h:409
#12 TObjArray::Streamer (this=0x559c85ba6410, b=...) at /usr/local/src/root/root-6.16.00/core/cont/src/TObjArray.cxx:486
#13 0x00007fe410bcc665 in TClass::Streamer (onfile_class=0x0, b=..., obj=0x559c85ba6410, this=0x559c7ed1fa28) at /usr/local/src/root/root-6.16.00-build/include/TClass.h:568
#14 TBufferFile::WriteFastArray (streamer=0x0, n=1, cl=0x559c7ed1fa28, start=<optimized out>, this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:2256
#15 TBufferFile::WriteFastArray (this=0x7fe3f2164f08, start=<optimized out>, cl=0x559c7ed1fa28, n=1, streamer=<optimized out>) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:2243
#16 0x00007fe410e19cb1 in TStreamerInfo::WriteBufferAux<char**> (this=0x7fe3fd243c08, b=..., arr=
0x7fff04ce8bc0: 0x7fff04ce8bb8, compinfo=0x7fe3fd24bfe0, first=first
entry=0, last=last
entry=1, narr=1, eoffset=0, arrayMode=0) at /usr/local/src/root/root-6.16.00-build/include/TString.h:239
#17 0x00007fe410c81130 in TStreamerInfoActions::GenericWriteAction (buf=..., addr=<optimized out>, config=<optimized out>) at /usr/local/src/root/root-6.16.00/io/io/src/TStreamerInfoActions.cxx:192
#18 0x00007fe410bd3202 in TStreamerInfoActions::TConfiguredAction::operator() (this=0x7fe3fd2389c8, this=0x7fe3fd2389c8, object=0x559c85ba62c8, buffer=...) at /usr/local/src/root/root-6.16.00-build/include/TStreamerInfoActions.h:124
#19 TBufferFile::ApplySequence (obj=0x559c85ba62c8, sequence=..., this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3564
#20 TBufferFile::ApplySequence (obj=0x559c85ba62c8, sequence=..., this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3546
#21 TBufferFile::WriteClassBuffer (this=this
entry=0x7fe3f2164f08, cl=0x559c883785e8, pointer=pointer
entry=0x559c85ba62c8) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3533
#22 0x00007fe410a32f2f in TBranch::Streamer (this=0x559c85ba62c8, b=...) at /usr/local/src/root/root-6.16.00/tree/tree/src/TBranch.cxx:2767
#23 0x00007fe410bd2022 in TClass::Streamer (onfile_class=0x0, b=..., obj=0x559c85ba62c8, this=0x559c883785e8) at /usr/local/src/root/root-6.16.00-build/include/TClass.h:568
#24 TBufferFile::WriteObjectClass (this=0x7fe3f2164f08, actualObjectStart=0x559c85ba62c8, actualClass=0x559c883785e8, cacheReuse=<optimized out>) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:2552
#25 0x00007fe410bd9e3d in TBufferIO::WriteObjectAny (cacheReuse=true, ptrClass=0x559c7e840028, obj=0x559c85ba62c8, this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferIO.cxx:519
#26 TBufferIO::WriteObjectAny (this=this
entry=0x7fe3f2164f08, obj=obj
entry=0x559c85ba62c8, ptrClass=0x559c7e840028, cacheReuse=cacheReuse
entry=true) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferIO.cxx:492
#27 0x00007fe41116c0a0 in operator<< (obj=0x559c85ba62c8, buf=...) at /usr/local/src/root/root-6.16.00-build/include/TBuffer.h:409
#28 TObjArray::Streamer (this=0x559c84222fc0, b=...) at /usr/local/src/root/root-6.16.00/core/cont/src/TObjArray.cxx:486
#29 0x00007fe410bcc665 in TClass::Streamer (onfile_class=0x0, b=..., obj=0x559c84222fc0, this=0x559c7ed1fa28) at /usr/local/src/root/root-6.16.00-build/include/TClass.h:568
#30 TBufferFile::WriteFastArray (streamer=0x0, n=1, cl=0x559c7ed1fa28, start=<optimized out>, this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:2256
#31 TBufferFile::WriteFastArray (this=0x7fe3f2164f08, start=<optimized out>, cl=0x559c7ed1fa28, n=1, streamer=<optimized out>) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:2243
#32 0x00007fe410e19cb1 in TStreamerInfo::WriteBufferAux<char**> (this=0x7fe3f2209d08, b=..., arr=
0x7fff04ce8f80: 0x7fff04ce8f78, compinfo=0x7fe3fd23e5a0, first=first
entry=0, last=last
entry=1, narr=1, eoffset=0, arrayMode=0) at /usr/local/src/root/root-6.16.00-build/include/TString.h:239
#33 0x00007fe410c81130 in TStreamerInfoActions::GenericWriteAction (buf=..., addr=<optimized out>, config=<optimized out>) at /usr/local/src/root/root-6.16.00/io/io/src/TStreamerInfoActions.cxx:192
#34 0x00007fe410bd3202 in TStreamerInfoActions::TConfiguredAction::operator() (this=0x7fe3fd239aa8, this=0x7fe3fd239aa8, object=0x559c84222e68, buffer=...) at /usr/local/src/root/root-6.16.00-build/include/TStreamerInfoActions.h:124
#35 TBufferFile::ApplySequence (obj=0x559c84222e68, sequence=..., this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3564
#36 TBufferFile::ApplySequence (obj=0x559c84222e68, sequence=..., this=0x7fe3f2164f08) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3546
#37 TBufferFile::WriteClassBuffer (this=this
entry=0x7fe3f2164f08, cl=0x559c842234c8, pointer=pointer
entry=0x559c84222e68) at /usr/local/src/root/root-6.16.00/io/io/src/TBufferFile.cxx:3533
#38 0x00007fe410aa48fc in TTree::Streamer (this=0x559c84222e68, b=...) at /usr/local/src/root/root-6.16.00/tree/tree/src/TTree.cxx:9216
#39 0x00007fe410c60087 in TMapFile::Update (this=0x7fe3f2164008, obj=obj
entry=0x0) at /usr/local/src/root/root-6.16.00/io/io/src/TMapFile.cxx:649
#40 0x0000559c7deabcca in QwRootFile::Update (this=0x559c7f9bbd48) at /home/wdconinc/git/japan/Analysis/include/QwRootFile.h:412
#41 QwRootFile::FillHistograms<QwSubsystemArrayParity> (this=this
entry=0x559c7f9bbd48, object=...) at /home/wdconinc/git/japan/Analysis/include/QwRootFile.h:333
#42 0x0000559c7dea4204 in main (argc=<optimized out>, argv=<optimized out>) at /home/wdconinc/git/japan/Parity/main/QwParity.cc:284
===========================================================

Segmentation fault (core dumped)

wdconinc commented 5 years ago

Putting the libNew even earlier (as first so, not merely first ROOT so) doesn't change anything either. I'm wondering if we can start from a standalone compiled TMapFile demonstration and verify that it works. Maybe we can also see if TMapFile has some diagnostics we can call before Update.

paulmking commented 5 years ago

@wdconinc If we look back at #26 , I seem to have thought that the engine at that time was able to handle the TMapFIle until the end of the run, instead of crashing immediately or in the middle of the run. So we could try an old version of the engine (end of December) to see if that is really "working". If so, maybe that helps point the way for the current engine.

wdconinc commented 5 years ago

Even hprod.C doesn't compile when adding -lNew...

13:31:47 wdconinc@herakles ~/git/japan (bugfix-using-TMapFile *$%=) $ g++ hprod.C -o hprod -std=c++11 -m64 -I/usr/local/root/pro/include -L/usr/local/root/pro/lib -lNew -lCore -lHist  -lMathCore -lRIO
13:32:05 wdconinc@herakles ~/git/japan (bugfix-using-TMapFile *$%=) $ ./hprod 
Error in <TMapFile::TMapFile>: no memory mapped file capability available
Use rootn.exe or link application against "-lNew"
Memory mapped file:   hsimple.map
Title:                Demo memory mapped file with histograms
Option:               file closed

wdconinc commented 5 years ago

duh

13:55:54 wdconinc@herakles ~/git/japan (bugfix-using-TMapFile *$%=) $ ldd hprod
        linux-vdso.so.1 (0x00007ffd2d5ec000)
        libCore.so => /usr/local/root/pro/lib/libCore.so (0x00007f57268a2000)
        libHist.so => /usr/local/root/pro/lib/libHist.so (0x00007f57264c4000)
        libMathCore.so => /usr/local/root/pro/lib/libMathCore.so (0x00007f57262b1000)
        libRIO.so => /usr/local/root/pro/lib/libRIO.so (0x00007f5725f11000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5725d87000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5725d16000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5725b2a000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f5725ab6000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f5725890000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5725673000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f572566d000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f572564c000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f57254bd000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f5726d54000)
        libMatrix.so => /usr/local/root/pro/lib/libMatrix.so (0x00007f5725340000)
        libImt.so => /usr/local/root/pro/lib/libImt.so (0x00007f572532d000)
        libThread.so => /usr/local/root/pro/lib/libThread.so (0x00007f57252d7000)
        libtbb.so.2 => /usr/local/root/pro/lib/libtbb.so.2 (0x00007f5725295000)

There's not libNew there at all... even though specified in the linker line.

wdconinc commented 5 years ago

LD_PRELOAD=/usr/local/root/pro/lib/libNew.so helps getting hprod and hcons working. No luck on qwparity, though. Still appears to be a TStreamer on WriteObjectAny.

wdconinc commented 5 years ago

./build/qwparity --enable-mapfile -r 10 --config qwparity_simple.conf --disable-trees --detectors mock_detectors.map is working for me (runs to completion). Problem seems to be with a tree.

wdconinc commented 5 years ago

Ok got it to finish running that test case without seg fault (disclaimer below).

Processing event 10000 (2.117 ms per event with 99.7978% efficiency)
Memory mapped file:   /dev/shm//QwMemMapFile.map
Title:                RealTime Producer File
Option:               CREATE
Mapped Memory region: 0x7f3765449000 - 0x7f37e5449000 (2048.00 MB)
Current breakval:     0x7f37ae0ab000
Normal end of file /home/wdconinc/qweak/QwScratch/data/QwMock_10.log encountered
Number of events processed at end of run: 0

(I put a TMapFile::Print() call in the QwRootFile::Update() for more output)

0x7f37ae0ab000 - 0x7f3765449000 = 0x48c62000 into the block, which is (well) over the 256 MiB default. I had to bump it up to 0x7fffffff (2GiB -1, 2 GiB failed for some reason). So, is there just a memory leak? Does it not delete the old TMapFile contents and keep overwriting new TTree cycles?

Disclaimer: it crashes at the end when cleaning up.

I have done everything I can do...

 *** Break *** segmentation violation

wdconinc commented 5 years ago

That was with

const Int_t QwRootFile::kMaxMapFileSize = 0x7fffffff; // 2 GiB

JeffersonLab / japan

Execution of "enable-mapfile" option results in crashes #69

Environment: (where does this bug occur, have you tried other environments)

Steps to reproduce: (give a step by step account of how to trigger the bug)