Crash with `DETECTOR_CONFIG=epic_craterlake_tracking_only`

veprbl commented 4 weeks ago

Environment: (where does this bug occur, have you tried other environments)

Which branch (often main for latest released):
Which version (or HEAD for the most recent on git): 1.14.1
Any specific OS or system where the issue occurs?
Any special versions of ROOT or Geant4?

Steps to reproduce: (give a step by step account of how to trigger the bug)

singularity shell -B /gpfs -B /gpfs02 docker://eicweb/eic_xl:24.06.0-stable-default
source /opt/detector/epic-24.06.0/bin/thisepic.sh && DETECTOR_CONFIG=epic_craterlake_tracking_only eicrecon /gpfs/mnt/gpfs02/eic/mposik/ePIC/test/ddsim/rootfiles/npsim_24.06.0_test_pi-_135.0deg_178.0deg_1.edm4eic.root

Expected Result: (what do you expect when you execute the steps above)

No crash

Actual Result: (what do you get when you execute the steps above)

Thread model: pthreads
140655485462208: 
       `- JSignalHandler::handle_sigsegv(int, siginfo_t*, void*) (0x7fed071e5302)
        `- /usr/lib/x86_64-linux-gnu/libc.so.6 (0x7fed06803050)
         `- AlgorithmsInit_service::acquire_services(JServiceLocator*)::{lambda(auto:1&&)#2}::operator()<algorithms::LogSvc&>(algorithms::LogSvc&) const::{lambda(algorithms::LogLevel, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >)#1}::operator()(algorithms::LogLevel, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >) const (0x7fed029588e5)
          `- void algorithms::LoggerMixin::report_fmt<(algorithms::LogLevel)4, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(fmt::v9::basic_format_string<char, fmt::v9::type_identity<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>::type>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) const (0x7fed0263b17b)
           `- /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/eicrecon-1.14.1-7ybfjmsg7q2mxkl66m7ybyigjeslouer/lib/libalgorithms_fardetectors.so (0x7fecebfda6b2)
            `- JMultifactory::Execute(std::shared_ptr<JEvent const> const&) (0x7fed0712b4bd)
             `- JFactory::Create(std::shared_ptr<JEvent const> const&) (0x7fed071245c0)
              `- JFactoryPodioT<edm4hep::TrackerHit>::Create(std::shared_ptr<JEvent const> const&) (0x7fece88e0a79)
               `- JEvent::GetCollectionBase(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool) const (0x7fece86cfe6a)
                `- JEventProcessorPODIO::Process(std::shared_ptr<JEvent const> const&) (0x7fece86c8824)
                 `- JEventProcessor::DoReduce(std::shared_ptr<JEvent const> const&) (0x7fed0716a5b0)
                  `- JEventProcessorArrow::process(std::shared_ptr<JEvent>*, bool&, JArrowMetrics::Status&) (0x7fed07140755)
                   `- JPipelineArrow<JEventProcessorArrow, std::shared_ptr<JEvent> >::execute(JArrowMetrics&, unsigned long) (0x7fed07145de9)
                    `- JWorker::loop() (0x7fed0713b9f9)
                     `- /opt/software/linux-debian12-x86_64_v2/gcc-12.2.0/gcc-runtime-12.2.0-dzlu7k7eoaoh4roeg24tzo3wilmeoyfz/lib/libstdc++.so.6 (0x7fed06a9c4a3)
                      `- /usr/lib/x86_64-linux-gnu/libc.so.6 (0x7fed06850134)
                       `- __clone (0x7fed068cfa40)

simonge commented 4 weeks ago

Looks slightly different to what Matt has, but based on both. Solution could be checking the readout exists before trying to create the factory as was done in tracking.cc for the various configurations. Swapping the error for a warning might also stop it from crashing?

veprbl commented 4 weeks ago

But can we also check for the defect that causes the crash instead of a completely disabled factory?

veprbl commented 4 weeks ago

Maybe enable this on github CI, there we have ASAN enabled. Or try locally with cmake -DUSE_ASAN=ON?

Simple-Shyam commented 1 week ago

Dear All, I am getting the following errors.

[FOFFMTRK:ForwardOffMRecParticles] [error] No beam protons to choose matrix!! Skipping!! [RPOTS:ForwardRomanPotRecParticles] [error] No beam protons to choose matrix!! Skipping!! [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/drich.lut" [PIDLookupTableSvc] [error] PID lookup table "calibrations/drich.lut" not found [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/drich.lut" [PIDLookupTableSvc] [error] PID lookup table "calibrations/drich.lut" not found [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/pfrich.lut" [PIDLookupTableSvc] [error] PID lookup table "calibrations/pfrich.lut" not found [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/pfrich.lut" [PIDLookupTableSvc] [error] PID lookup table "calibrations/pfrich.lut" not found [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/drich.lut" [PIDLookupTableSvc] [error] PID lookup table "calibrations/drich.lut" not found [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/drich.lut" [PIDLookupTableSvc] [error] PID lookup table "calibrations/drich.lut" not found [tracking:CentralCKFSeededTrajectories] [info] Last trajectory is empty [tracking:CentralCKFSeededTrajectories] [warning] Empty multiTrajectory. [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/tof.lut" [PIDLookupTableSvc] [error] PID lookup table "calibrations/tof.lut" not found [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/tof.lut" [PIDLookupTableSvc] [error] PID lookup table "calibrations/tof.lut" not found [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/hpdirc_positive.lut" [PIDLookupTableSvc] [error] PID lookup table "calibrations/hpdirc_positive.lut" not found [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/hpdirc_positive.lut" [PIDLookupTableSvc] [error] PID lookup table "calibrations/hpdirc_positive.lut" not found [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/drich.lut" [PIDLookupTableSvc] [error] PID lookup table "calibrations/drich.lut" not found [PIDLookupTableSvc] [info] Loading PID lookup table "calibrations/drich.lut"

How to disable these? Is someone knows it's making reconstruction very slow.

veprbl commented 1 week ago

The way to make it work is to use recent enough epic geometry version that provides those LUT files.

The way to disable it is to set -Ptracking:ChargedParticlesWithAssociations:OutputTags=ReconstructedChargedParticles,ReconstructedChargedParticleAssociations -PDRICH:DRICHLUTPID:OutputTags=unused1,unused2, I believe.

Simple-Shyam commented 1 week ago

Thanks @veprbl. The things are working fine with recent geometry. As I mentioned that the tracking performances for large eta region some nonphysical behavior so I am running locally with all the older epic detector version with the recent EICRecon, because older EICRecon wasn't working because of edm data format is updated to new in the container.
For running older tag locally @ShujieL eic container, epic, and EICRecon must be compatible. In this case, I can check the simulation campaign to check the older results and also disable flag as you suggested.

eic / EICrecon