Closed wausain closed 5 years ago
@wausain, can you list the commands you used to run flreconstruct
please? If that included generation/use of a simulation file, please also let us know how to generate/obtain that.
flsimulate script:
_#@format=datatools::configuration::variant
[name="flsimulate" type="flsimulate::section"] numberOfEvents : integer = 100
[name="geometry"] layout = "Basic" layout/if_basic/magnetic_field = true layout/if_basic/magnetic_field/is_active/type = "UniformVertical" layout/if_basic/magnetic_field/is_active/type/if_uniform_vertical/magnitude = 25 gauss layout/if_basic/magnetic_field/is_active/type/if_uniform_vertical/direction = "+z" layout/if_basic/source_layout = "Basic" layout/if_basic/source_layout/if_basic/thickness = 250 um layout/if_basic/source_layout/if_basic/material = "Se82" layout/if_basic/source_calibration = false layout/if_basic/shielding = true calo_film_thickness = 25 um
[name="vertexes"] generator = "source_pads_bulk"
[name="primary_events"] generator = "Se82.0nubb"
[name="simulation"] physics_mode = "Constructors" physics_mode/if_constructors/em_model = "standard" production_cuts = true outputprofile = "none"
flreconstruct script:
_#@key_label "name"
[name="flreconstruct.plugins" type="flreconstruct::section"] plugins : string[6] = "Falaise_CAT" \ "TrackFit" \ "Falaise_TrackFit" \ "Falaise_ChargedParticleTracking" \ "GammaTracking" \ "Falaise_GammaClustering" [name="pipeline" type="dpp::chain_module"] modules : string[6] = \ "CalibrateTracker" \ "CalibrateCalorimeters" \ "CATTrackerClusterizer" \ "TrackFitting" \ "ChargedParticleTracking" \ "GammaClusterizer"
[name="CalibrateTracker" type="snemo::processing::mock_tracker_s2c_module"] [name="CalibrateCalorimeters" type="snemo::processing::mock_calorimeter_s2c_module"] [name="CATTrackerClusterizer" type="snemo::reconstruction::cat_tracker_clustering_module"] TPC.processing_prompt_hits : boolean = 1 TPC.processing_delayed_hits : boolean = 1 [name="TrackFitting" type="snemo::reconstruction::trackfit_tracker_fitting_module"] fitting_models : string[2] = "helix" "line" line.only_guess : string[4] = "BB" "BT" "TB" "TT" helix.only_guess : string[8] = "BBB" "BBT" "BTB" "BTT" "TBB" "TBT" "TTB" "TTT"
line.guess.fit_delayed_clusters : boolean = 1 [name="ChargedParticleTracking" type="snemo::reconstruction::charged_particle_tracking_module"]
AFD.minimal_delayed_time : real as time = 10 us
AFD.minimal_cluster_xy_search_distance : real as length = 40 cm [name="GammaClusterizer" type="snemo::reconstruction::gamma_clustering_module"]
Geo_label : string = "geometry"
I've changed the no of events in the simulation and the error ceases to exist when the no of events are <~10.
Just to clarify - does it always crash on the same event? If you have a different input file, does it still crash on the same event number (even if the events are not the same)? And @drbenmorgan - quick question - can you remember whether the random number seed in flsimulate is constant if you don't specify it? So if you ran flsimulate twice with the same config, and didn't specify a seed, would you get the same simulation out? I know we discussed it in the past but I can't remember what was decided.
@cherylepatrick It doesn't look to crash on a specific event, it just crashes and fails to run.
From the terminal output:
_[notice:void datatools::library_loader::init():450] Automatic loading of library 'Falaise_CAT'... [notice:void datatools::library_loader::init():450] Automatic loading of library 'TrackFit'... [notice:void datatools::library_loader::init():450] Automatic loading of library 'Falaise_TrackFit'... [notice:void datatools::library_loader::init():450] Automatic loading of library 'Falaise_ChargedParticleTracking'... [notice:void datatools::library_loader::init():450] Automatic loading of library 'GammaTracking'... [notice:void datatools::library_loader::init():450] Automatic loading of library 'Falaise_GammaClustering'... CAT::clusterizer::initialize: Entering... CAT::clusterizer::readDstProper: Entering... CAT::clusterizer::readDstProper: SuperNemo kind of data CAT::clusterizer::readDstProper: small radius 2 mm CAT::clusterizer::readDstProper: tangent phi 20 CAT::clusterizer::readDstProper: tangent theta 160 CAT::clusterizer::readDstProper: small number 0.1 mm CAT::clusterizer::readDstProper: quadrant angle 90 CAT::clusterizer::readDstProper: ratio 10000 CAT::clusterizer::readDstProper: compatibility distance 4 CAT::clusterizer::readDstProper: maximum chi2 3 CAT::clusterizer::readDstProper: probmin 0 CAT::clusterizer::readDstProper: first event number -1 CAT::clusterizer::readDstProper: cell pitch: 43.95 mm CAT::clusterizer::readDstProper: verbosity print level: 2 CAT::clusterizer::readDstProper: Done. CAT::clusterizer::initialize: Done. CAT::sequentiator::readDstProper: SuperNemo kind of data
Break segmentation violation_
Thanks @wausain, @cherylepatrick, I've been able to reproduce the error, and it looks to be the same "EXC_BAD_ACCESS" error we've seen in gamma tracking/clustering (but seemingly not from that). It also seems to happen in the default setup as well.
I'll post what I have so far here, but this is going to need some reinstall (it's possible something on macOS system or brew libs has changed) and local debug builds to trace it fully. So far, I just ran:
flsimulate
with the default setup, 100 events:
#@key_label "name"
#@meta_label "type"
[name="flsimulate" type="flsimulate::section"]
numberOfEvents : integer = 100
so running as:
$ flsimulate -p default_sim.conf -o default.sim.brio
Put the output through the default repo pipeline:
$ flreconstruct -V trace -i default.sim.brio -p urn:snemo:demonstrator:reconstruction:1.0.0 -o default.reco.brio
Events are processed, and the seg fault happens (which event it occurs on is deterministic, albeit dependent on the random number seeding used to generate it). Running in lldb:
$ lldb -- flreconstruct -V trace -i default.sim.brio -p urn:snemo:demonstrator:reconstruction:1.0.0 -o default.reco.brio
(lldb) target create "flreconstruct"
Current executable set to 'flreconstruct' (x86_64).
(lldb) settings set -- target.run-args "-V" "trace" "-i" "default.sim.brio" "-p" "urn:snemo:demonstrator:reconstruction:1.0.0" "-o" "default.reco.brio"
(lldb) run
... output, then...
[trace:base_module::process_status dpp::input_module::_load(datatools::things &):370] Entering...
[trace:base_module::process_status dpp::input_module::_load(datatools::things &):373] filenames.size() = 1
Process 53243 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
frame #0: 0x0000000100180b04 libFalaise.3.dylib`void boost::archive::detail::save_pointer_type<eos::portable_oarchive>::polymorphic::save<snemo::datamodel::tracker_clustering_solution>(eos::portable_oarchive&, snemo::datamodel::tracker_clustering_solution&) + 46
libFalaise.3.dylib`boost::archive::detail::save_pointer_type<eos::portable_oarchive>::polymorphic::save<snemo::datamodel::tracker_clustering_solution>:
-> 0x100180b04 <+46>: movq (%r15), %rax
0x100180b07 <+49>: movq -0x8(%rax), %rsi
0x100180b0b <+53>: movq %r12, %rdi
0x100180b0e <+56>: callq 0x1002b3c8a ; symbol stub for: boost::serialization::typeid_system::extended_type_info_typeid_0::get_extended_type_info(std::type_info const&) const
Target 0: (flreconstruct) stopped.
Thus appears to be null/bad memory, so need to track down why that occurs. It also happens outputting xml instead of brio/root...
Possibly related to:
Thanks @drbenmorgan . Appreciate your help - these segfaults are a nightmare.
O.k., I've reinstalled Falaise from scratch on current macOS/Xcode and the problem persists, so it is not due to macOS or brew issue. More to follow...
Tried reinstalling both Bayeux and Falaise in full Debug mode, same effect, no additional info so trying a debug build locally.
Right, local development build of Falaise with the 3.3.0 tag, plus standard brew install of Bayeux (so that's in Release, optimised, mode). Building Falaise with CMAKE_BUILD_TYPE
as Debug
and running the same scripts as above, I cannot reproduce the error. Changing the build mode to RelWithDebInfo
(or Release
) reproduces the error, with the errorr now being:
Process 62995 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
frame #0: 0x00000001001a54fe libFalaise.3.dylib`void boost::archive::detail::save_pointer_type<boost::archive::xml_oarchive>::polymorphic::save<snemo::datamodel::tracker_clustering_solution>(boost::archive::xml_oarchive&, snemo::datamodel::tracker_clustering_solution&) [inlined] boost::serialization::extended_type_info_typeid<snemo::datamodel::tracker_clustering_solution>::get_derived_extended_type_info(this=0x00000001003ffdd8, t=0x0000000000000000) const at extended_type_info_typeid.hpp:107 [opt]
104 BOOST_STATIC_WARNING(boost::is_polymorphic< T >::value);
105 return
106 typeid_system::extended_type_info_typeid_0::get_extended_type_info(
-> 107 typeid(t)
108 );
109 }
110 const char * get_key() const {
Target 0: (flreconstruct) stopped.
Full backtrace in the attached file: bt.txt It can be seen that the t
reference passed to the boost function is null:
* frame #0: 0x00000001001a5e0e libFalaise.3.dylib`void boost::archive::detail::save_pointer_type<eos::portable_oarchive>::polymorphic::save<snemo::datamodel::tracker_clustering_solution>(eos::portable_oarchive&, snemo::datamodel::tracker_clustering_solution&) [inlined] boost::serialization::extended_type_info_typeid<snemo::datamodel::tracker_clustering_solution>::get_derived_extended_type_info(this=0x00000001003ffdd8, t=0x0000000000000000) const at extended_type_info_typeid.hpp:107 [opt]
Given the apparent functioning in Debug mode, I would suspect something is getting optimised out, as in BxCppDev/Bayeux#20.
All I can recommend for now is to build Falaise yourself locally in Debug mode, i.e. build as normal, but add -DCMAKE_BUILD_TYPE=Debug
to the cmake command line arguments.
@fmauger, could you take a look at this and provide a fix please as it is heavily coupled to Bayeux/Boost serialisation?
Thank you @drbenmorgan I've managed to install in Debug mode and it seems to be working fine so far.
A little further info, I've tried an experimental build with a newer version of Boost (1.66) and C++14. Whilst this is a little hacky (doesn't include flsimulate yet, Boost doesn't use icu4c), it can read and run the default pipeline over yesterday's simulation output in Release/RelWithDebInfo modes. That could suggest an issue in our production Boost version (1.63) that the newest Clang compiler is picking up.
I'm not going to suggest an immediate upgrade as we'll need to validate that updating the Boost version won't cause any compatibility issues reading older Boost 1.63 generated files. If any macOS user wants to try this out, ping me here or on the Software Slack channel and I'll outline the steps needed.
Hiya, I finally got round to trying this but I'm having trouble make
-ing Falaise. Was doing OK until I got up to Things2Root:
Scanning dependencies of target Things2Root
[ 98%] Building CXX object modules/things2root/CMakeFiles/Things2Root.dir/Things2Root.cpp.o
In file included from /Users/cpatrick/SuperNEMO/github/Falaise.git/modules/things2root/Things2Root.cpp:9:
In file included from /Users/cpatrick/SuperNEMO/github/Falaise.git/modules/things2root/Things2Root.h:15:
In file included from /Users/cpatrick/brew/include/bayeux/datatools/service_manager.h:46:
In file included from /Users/cpatrick/brew/include/bayeux/datatools/base_service.h:44:
/Users/cpatrick/brew/include/bayeux/datatools/enriched_base.h:197:5: fatal error: 'get_serial_tag' overrides a member function but
is not marked 'override' [-Winconsistent-missing-override]
DATATOOLS_SERIALIZATION_DECLARATION_ADVANCED(enriched_base)
^
/Users/cpatrick/brew/include/bayeux/datatools/i_serializable.h:372:3: note: expanded from macro
'DATATOOLS_SERIALIZATION_DECLARATION_ADVANCED'
DATATOOLS_SERIALIZATION_DECLARATION() \
^
/Users/cpatrick/brew/include/bayeux/datatools/i_serializable.h:268:3: note: expanded from macro
'DATATOOLS_SERIALIZATION_DECLARATION'
DATATOOLS_SERIALIZATION_SERIAL_TAG_DECLARATION() \
^
/Users/cpatrick/brew/include/bayeux/datatools/i_serializable.h:123:30: note: expanded from macro
'DATATOOLS_SERIALIZATION_SERIAL_TAG_DECLARATION'
virtual const std::string& get_serial_tag() const; \
^
/Users/cpatrick/brew/include/bayeux/datatools/i_serializable.h:63:33: note: overridden virtual function is here
virtual const std::string & get_serial_tag() const = 0;
^
1 error generated.
make[2]: *** [modules/things2root/CMakeFiles/Things2Root.dir/Things2Root.cpp.o] Error 1
make[1]: *** [modules/things2root/CMakeFiles/Things2Root.dir/all] Error 2
make: *** [all] Error 2
I don't necessarily mind if I can't use Things2Root, but since it crashed, it hasn't copied anything to my install directory.
This was the cmake command I used, in case I made a mistake there: cmake -DCMAKE_INSTALL_PREFIX=../FalaiseInstall/ -DCMAKE_BUILD_TYPE=Debug ../Falaise.git
Any ideas? Thanks in advance.
@cherylepatrick, it should be possible to ignore these by building Falaise with the option to promote warnings to errors off:
$ cmake -DFALAISE_COMPILER ERROR_ON_WARNING=OFF -DCMAKE_INSTALL_PREFIX=../FalaiseInstall/ -DCMAKE_BUILD_TYPE=Debug ../Falaise.git
I have tried to fix the issues in Bayeux, but ended up with a cascade failure of additional ones...
You might find that there are further issues with boost/bayeux/root, but first things first.
Footnote: Going to higher versions of Boost will be difficult at the moment if it's the ultimate cause of this issue. Bayeux fails a large number of tests with 1.68 on Linux/macOS, so I'm backtracking to find "last good version".
Thanks, @drbenmorgan, and sorry this is hassle. I tried what you suggested, but it didn't like it - do I have a typo?
_BUILD_TYPE=Debug ../Falaise.git
Cheryls-MBP:Falaise.build cpatrick$ cmake -DFALAISE_COMPILER ERROR_ON_WARNING=OFF -DCMAKE_INSTALL_PREFIX=../FalaiseInstall/ -DCMAKE_BUILD_TYPE=Debug ../Falaise.git
Parse error in command line argument: -DFALAISE_COMPILER
Should be: VAR:type=value
CMake Error: No cmake script provided.
CMake Error: Problem processing arguments. Aborting.
Oh I tried with an extra underscore and it's doing SOMETHING at least... -DFALAISECOMPILER****ERROR_ON_WARNING=OFF
Suspect this is fixed by the migration to Boost 1.69.
Brew doctor and brew update show no issues.
Bug reports:
Crashing during reconstruction (flreconstruct) but only for >~10 events. Seems to be caused by trackfit or charged particle tracking.
_HOMEBREW_VERSION: 1.6.9-27-g7db0f97 ORIGIN: https://github.com/Homebrew/brew HEAD: 7db0f97523415917625bd6136076092b5c4977dd Last commit: 20 hours ago Core tap ORIGIN: https://github.com/Homebrew/homebrew-core Core tap HEAD: 241f36d8e82b2aa6c3e016ac5807a15598023c23 Core tap last commit: 20 hours ago HOMEBREW_PREFIX: /usr/local HOMEBREW_DEV_CMD_RUN: 1 HOMEBREW_LINKAGE_CACHE: 1 CPU: quad-core 64-bit kabylake Homebrew Ruby: 2.3.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby Clang: 9.1 build 902 Git: 2.15.2 => /Applications/Xcode.app/Contents/Developer/usr/bin/git Curl: 7.54.0 => /usr/bin/curl Java: 10.0.1 macOS: 10.13.5-x8664 CLT: 9.4.1.0.1.1528165917 Xcode: 9.4.1 XQuartz: N/A
Brew Doctor: _Warning: "config" scripts exist outside your system or Homebrew directories.
./configure
scripts often look for *-config scripts to determine if software packages are installed, and what additional flags to use when compiling and linking.Having additional scripts in your path can confuse software installed via Homebrew if the config script overrides a system or Homebrew provided script of the same name. We found the following "config" scripts: /Users/hamzah/anaconda3/bin/icu-config /Users/hamzah/anaconda3/bin/freetype-config /Users/hamzah/anaconda3/bin/xslt-config /Users/hamzah/anaconda3/bin/libpng16-config /Users/hamzah/anaconda3/bin/python3.6m-config /Users/hamzah/anaconda3/bin/libpng-config /Users/hamzah/anaconda3/bin/xml2-config /Users/hamzah/anaconda3/bin/python3-config /Users/hamzah/anaconda3/bin/curl-config /Users/hamzah/anaconda3/bin/ncursesw6-config /Users/hamzah/anaconda3/bin/pcre-config /Users/hamzah/anaconda3/bin/python3.6-config_
No issues with the build (shall i still provide gist-logs?)
Error on crash:
_ Break segmentation violation [/usr/lib/system/libsystem_platform.dylib] _sigtramp (no debug info) [] (no debug info)
[/usr/local/Cellar/falaise/3.3.0/lib/libFalaise.3.3.0.dylib] boost::archive::detail::oserializer<eos::portable_oarchive, boost::shared_ptr >::save_object_data(boost::archive::detail::basic_oarchive&, void const) const (no debug info)
[/usr/local/opt/boost/lib/libboost_serialization-mt.dylib] boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::detail::basic_oarchive&, void const, boost::archive::detail::basic_oserializer const&) (no debug info)
[/usr/local/Cellar/falaise/3.3.0/lib/libFalaise.3.3.0.dylib] boost::archive::detail::oserializer<eos::portable_oarchive, datatools::handle >::save_object_data(boost::archive::detail::basic_oarchive&, void const) const (no debug info)
[/usr/local/opt/boost/lib/libboost_serialization-mt.dylib] boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::detail::basic_oarchive&, void const, boost::archive::detail::basic_oserializer const&) (no debug info)
[/usr/local/Cellar/falaise/3.3.0/lib/libFalaise.3.3.0.dylib] void snemo::datamodel::particle_track::serialize(eos::portable_oarchive&, unsigned int) (no debug info)
[/usr/local/Cellar/falaise/3.3.0/lib/libFalaise.3.3.0.dylib] boost::archive::detail::oserializer<eos::portable_oarchive, snemo::datamodel::particle_track>::save_object_data(boost::archive::detail::basic_oarchive&, void const) const (no debug info)
[/usr/local/opt/boost/lib/libboost_serialization-mt.dylib] boost::archive::detail::basic_oarchive_impl::save_pointer(boost::archive::detail::basic_oarchive&, void const, boost::archive::detail::basic_pointer_oserializer const) (no debug info)
[/usr/local/Cellar/falaise/3.3.0/lib/libFalaise.3.3.0.dylib] boost::archive::detail::oserializer<eos::portable_oarchive, boost::shared_ptr >::save_object_data(boost::archive::detail::basic_oarchive&, void const ) const (no debug info)
[/usr/local/opt/boost/lib/libboost_serialization-mt.dylib] boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::detail::basic_oarchive&, void const, boost::archive::detail::basic_oserializer const&) (no debug info)
[/usr/local/Cellar/falaise/3.3.0/lib/libFalaise.3.3.0.dylib] boost::archive::detail::oserializer<eos::portable_oarchive, datatools::handle >::save_object_data(boost::archive::detail::basic_oarchive&, void const ) const (no debug info)
[/usr/local/opt/boost/lib/libboost_serialization-mt.dylib] boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::detail::basic_oarchive&, void const, boost::archive::detail::basic_oserializer const&) (no debug info)
[/usr/local/Cellar/falaise/3.3.0/lib/libFalaise.3.3.0.dylib] boost::archive::detail::oserializer<eos::portable_oarchive, std::__1::vector<datatools::handle, std::__1::allocator<datatools::handle > > >::save_object_data(boost::archive::detail::basic_oarchive&, void const ) const (no debug info)
[/usr/local/opt/boost/lib/libboost_serialization-mt.dylib] boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::detail::basic_oarchive&, void const, boost::archive::detail::basic_oserializer const&) (no debug info)
[/usr/local/Cellar/falaise/3.3.0/lib/libFalaise.3.3.0.dylib] void snemo::datamodel::particle_track_data::serialize(eos::portable_oarchive&, unsigned int) (no debug info)
[/usr/local/Cellar/falaise/3.3.0/lib/libFalaise.3.3.0.dylib] boost::archive::detail::oserializer<eos::portable_oarchive, snemo::datamodel::particle_track_data>::save_object_data(boost::archive::detail::basic_oarchive&, void const ) const (no debug info)
[/usr/local/opt/boost/lib/libboost_serialization-mt.dylib] boost::archive::detail::basic_oarchive_impl::save_pointer(boost::archive::detail::basic_oarchive&, void const, boost::archive::detail::basic_pointer_oserializer const) (no debug info)
[/usr/local/opt/bayeux/lib/libBayeux.3.dylib] boost::archive::detail::oserializer<eos::portable_oarchive, datatools::things::entry_type>::save_object_data(boost::archive::detail::basic_oarchive&, void const) const (no debug info)
[/usr/local/opt/boost/lib/libboost_serialization-mt.dylib] boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::detail::basic_oarchive&, void const, boost::archive::detail::basic_oserializer const&) (no debug info)
[/usr/local/opt/bayeux/lib/libBayeux.3.dylib] boost::archive::detail::oserializer<eos::portable_oarchive, std::1::pair<std::1::basic_string<char, std::1::char_traits, std::__1::allocator > const, datatools::things::entry_type> >::save_object_data(boost::archive::detail::basic_oarchive&, void const) const (no debug info)
[/usr/local/opt/boost/lib/libboost_serialization-mt.dylib] boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::detail::basic_oarchive&, void const, boost::archive::detail::basic_oserializer const&) (no debug info)
[/usr/local/opt/bayeux/lib/libBayeux.3.dylib] boost::archive::detail::oserializer<eos::portable_oarchive, std:: 1::map<std::1::basic_string<char, std::__1::char_traits, std:: 1::allocator >, datatools::things::entry_type, std::1::less<std::1::basic_string<char, std::1::char_traits, std::1::allocator > >, std:: 1::allocator<std::1::pair<std::1::basic_string<char, std:: 1::char_traits, std::1::allocator > const, datatools::things::entry_type> > > >::save_object_data(boost::archive::detail::basic_oarchive&, void const) const (no debug info)
[/usr/local/opt/boost/lib/libboost_serialization-mt.dylib] boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::detail::basic_oarchive&, void const, boost::archive::detail::basic_oserializer const&) (no debug info)
[/usr/local/opt/bayeux/lib/libBayeux.3.dylib] boost::archive::detail::oserializer<eos::portable_oarchive, datatools::things>::save_object_data(boost::archive::detail::basic_oarchive&, void const) const (no debug info)
[/usr/local/opt/boost/lib/libboost_serialization-mt.dylib] boost::archive::detail::basic_oarchive_impl::save_object(boost::archive::detail::basic_oarchive&, void const, boost::archive::detail::basic_oserializer const&) (no debug info)
[/usr/local/opt/bayeux/lib/libBayeux.3.dylib] int brio::writer::_at_store(datatools::things const&, brio::store_info*) (no debug info)
[/usr/local/opt/bayeux/lib/libBayeux.3.dylib] int brio::writer::store(datatools::things const&, std::__1::basic_string<char, std:: 1::char_traits, std::__1::allocator > const&) (no debug info)
[/usr/local/opt/bayeux/lib/libBayeux.3.dylib] dpp::simple_brio_data_sink::store_next_record(datatools::things const&) (no debug info)
[/usr/local/opt/bayeux/lib/libBayeux.3.dylib] dpp::output_module::_store(datatools::things const&) (no debug info)
[/usr/local/bin/flreconstruct] FLReconstruct::do_pipeline(FLReconstruct::FLReconstructParams const&) (no debug info)
[/usr/local/bin/flreconstruct] FLReconstruct::doflreconstruct(int, char**) (no debug info)
[/usr/local/bin/flreconstruct] main (no debug info)
[/usr/lib/system/libdyld.dylib] start (no debug info)
Segmentation fault: 11
Error seems to be caused by the track fit or charged particle tracking modules in the pipeline.