Can't read enough events from evio files

DraTeots commented 1 year ago

Sergey provided us what evio files in different configurations could be used as samples:

Physics runs:
-------------

===>  2 crates , 3 detectors : CAL/FA250, GEMTRD/FA125 , GEM/SRS

CODA::  Run_2531 GEMTRD:ok; CAL:ok; SRS:del=0x41; ROSYBC=0x90 3bin; 10APV;OK; TRDpos=100; 5.1M ev  *PHYS* !!!!
cDAQ::  Run_2543 GEMTRD:ok; CAL:ok; SRS:del=0x41; ROSYBC=0x90 3bin; 10APV;OK; TRDpos=100; 1.1M ev  *PHYS* !!!!

===> 1 crate ; 2 detectors : GEMTRD/FA125 ; GEM/SRS

CODA::  Run_2548 (CODA) GEMTRD:ok; CAL:No; SRS:del=0x40; ROSYBC=0x70 9bin; 10APV;OK; TRDpos=155; 1.5M ev  *PHYS* !!!!
cDAQ::  Run_2567 (cDAQ) GEMTRD:ok; CAL:No; SRS:del=0x40; ROSYBC=0x70 9bin; 10APV;OK; TRDpos=155; 3.2M ev  *PHYS* !!!!

I run jana in this configuration for all of them:

jana
-Pplugins=log,root_output,single_event_evio,example_evio_analysis
-Pjana:debug_plugin_loading=1
-Pcdaq:LogLevel=trace
-Pjana:timeout=0
-Pjana:nevnets=1
-Pnthreads=1
/mnt/work/data/2023-03-03-trd-data/hd_rawdata_002543_000.evio

rawdata_002531_000.evio

CODA::  Run_2531 GEMTRD:ok; CAL:ok; SRS:del=0x41; ROSYBC=0x90 3bin; 10APV;OK; TRDpos=100; 5.1M ev  *PHYS* !!!!

There are 3 kind of problems:

Sometimes (the rarest case) it reads file, but doesn't find any data and always complain:

Event number = 1
  Factory = DCODAControlEvent NumObjects = 1
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
Event number = 2
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
Event number = 3
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
Event number = 4
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
Event number = 5
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
Event number = 6

Sometimes it writes and quits:

jana: /home/romanov/eic/soft/jana2/src/master/src/libraries/JANA/Utils/JEventPool.h:38: JEventPool::JEventPool(std::shared_ptr<JComponentManager>, size_t, size_t, bool): Assertion `m_pool_size > 0 || !m_limit_total_events_in_flight' failed.

Sometimes it initializes OK but then if plugin throws an exception, and everything goes off the rails. It start spitting raw binary data to screen (corruption of something that cout-ed?)

Event number = 42
[INFO] JWorker: Worker 5464 shutdown due to JException: Could not find JFactoryT<DGEMSRSWindowRawData> with tag=
[INFO] JWorker: Worker 2073 shutdown due to JException: Could not find JFactoryT<DGEMSRSWindowRawData> with tag=
[INFO] JWorker: Worker 3932 shutdown due to JException: Could not find JFactoryT<DGEMSRSWindowRawData> with tag=
[INFO] JWorker: Worker 5500 shutdown due to JException: Could not find JFactoryT<DGEMSRSWindowRawData> with tag=
[INFO] JWorker: Worker 5753 shutdown due to JException: Could not find JFactoryT<DGEMSRSWindowRawData> with tag=
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
Event number = 43
[INFO] JWorker: Worker 4629 shutdown due to JException: Could not find JFactoryT<DGEMSRSWindowRawData> with tag=
[INFO] JWorker: Worker 3315 shutdown due to JException: Could not find JFactoryT<DGEMSRSWindowRawData> with tag=
[INFO] JWorker: Worker 6238 shutdown due to JException: Could not find JFactoryT<DGEMSRSWindowRawData> with tag=
...
----------------------------------------------------------
Exception: Resource temporarily unavailable
[2023-03-08 12:30:10.374] [RootFile] [info] Closed user histogram file: eicrecon.root
/home/romanov/eic/JANA4ML4FPGA/src/libraries/evio/HDEVIO.cc:106 Call HDEVIO destructor!UA�P�! 
OK��z�LK��zb�PK��zb�S�L   Ux, ...

And spits like gygabytes of symbols per second!!!

Guess this one is the key:

#ifdef USE_ASYNC_FILEBUF
    _DBG_ << "Call HDEVIO destructor!" << ifs.std::ios::rdbuf() << std::endl;
//    delete ifs.std::ios::rdbuf();   /// @davidl
    ifs.std::ios::rdbuf(ifs.rdbuf());
#endif

Since this case is very insteresting here is the log (the rest 2GB is truncated) jana-read-error-log.txt

hd_rawdata_002543_000.evio

cDAQ::  Run_2543 GEMTRD:ok; CAL:ok; SRS:del=0x41; ROSYBC=0x90 3bin; 10APV;OK; TRDpos=100; 1.1M ev  *PHYS* !!!!

prints periodically:

Event number = 965
...
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:156 Unknown outer EVIO bank tag: 0
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:156 Unknown outer EVIO bank tag: 0
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:156 Unknown outer EVIO bank tag: 0
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:156 Unknown outer EVIO bank tag: 0
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:156 Unknown outer EVIO bank tag: 0
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:156 Unknown outer EVIO bank tag: 0
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:156 Unknown outer EVIO bank tag: 0
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:156 Unknown outer EVIO bank tag: 0
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:156 Unknown outer EVIO bank tag: 0

DraTeots commented 1 year ago

Continue, so:

rawdata_002548_000.evio

CODA::  Run_2548 (CODA) GEMTRD:ok; CAL:No; SRS:del=0x40; ROSYBC=0x70 9bin; 10APV;OK; TRDpos=155; 1.5M ev  *PHYS* !!!!

Same not working.

[2023-03-08 13:27:57.396] [SingleEvioEventFileSource] [debug] SingleEvioEventFileSource::GetEvent
[2023-03-08 13:27:57.398] [SingleEvioEventFileSource] [debug] Block 15 read_ok!
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event
/home/romanov/eic/JANA4ML4FPGA/src/libraries/rawdataparser/EVIOBlockedEventParser.cc:278 Skipping Physics event

It doesn't fine any data in the events

DraTeots commented 1 year ago

yes, after commeting out

#ifdef USE_ASYNC_FILEBUF
    _DBG_ << "Call HDEVIO destructor!" << ifs.std::ios::rdbuf() << std::endl;
//    delete ifs.std::ios::rdbuf();   /// @davidl
    ifs.std::ios::rdbuf(ifs.rdbuf());
#endif

It stopped spitting gigabytes in console =)

cissieAB commented 1 year ago

With "CAQfile" plugin

I have similar behavior for 2531 and 2548
I have no "Unknown outer EVIO bank tag: 0" for 2543 but is seems never end

DraTeots commented 1 year ago

All problems except "Unknown outer EVIO bank tag: 0" are fixed. The later doesn't seem like a blocking problem at the moment.

JeffersonLab / JANA4ML4FPGA