Small values for HSI and Trigger app request_timeout do not seem to have the expected effect in v5

[bieryAtFnal]

In a small v5 test system, I noticed that HSI and Trigger fragments are not arriving at the TriggerRecordBuilder in a timely way.

With a request timeout of 1 second or more, and with a trigger rate of approximately 1 Hz, this may not be surprising. The reason for this is that the data-handler code waits for a subsequent data packet before sending out a fragment with an initial data packet in it.

However, we would expect that a small request timeout value, say 10 msec, would help the HSI and Trigger fragments show up at the TRB sooner than the next trigger (i.e. 1000 msec == 1 sec).

Setting the request timeout to zero does produce the desired behavior.

Here are sample instructions for demonstrating the observed behavior. The first run an (approximately) out-of-the-box system based on the sample config in the appmodel repo. Then, they set the HSI and Trigger request timeouts to 10 msec. Then, they set them to zero.

TRACE messages are used to demonstrate the behavior, and I'll post a sample TRACE output next.

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest_v5
dbt-create -n NFD_DEV_240918_A9 ${DATE_PREFIX}FDv5Test_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDv5Test_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/appmodel.git -b develop
cd appmodel; git checkout 680f18b515; cd ..
cd ..

dbt-workarea-env
dbt-build -j 12
dbt-workarea-env

mkdir -p rundir/demo_configs

if [[ ! -d rundir/demo_configs/config ]]; then
   cp -Rv sourcecode/appmodel/config rundir/demo_configs/.
fi
if [[ ! -d rundir/demo_configs/test ]]; then
   cp -Rv sourcecode/appmodel/test rundir/demo_configs/.
   cp rundir/demo_configs/test/config/test-session.data.xml rundir/demo_configs/test/config/demo-configs.data.xml
   sed -i 's/test-session/simple-demo-system/'g rundir/demo_configs/test/config/demo-configs.data.xml
fi

if [[ ! -e local_setup ]]; then
cat <<EOF1 > local_setup
if [[ "\`echo \${DUNEDAQ_DB_PATH} | grep demo_configs\`" == "" ]] ; then
   export DUNEDAQ_DB_PATH=\${PWD}/rundir/demo_configs:\${DUNEDAQ_DB_PATH}
fi
export TRACE_FILE=${PWD}/log/${USER}_dunedaq.trace
EOF1
fi
. ./local_setup

oks_enable_tpg -d -s simple-demo-system ./rundir/demo_configs/test/config/demo-configs.data.xml 
oks_enable -d -s simple-demo-system ./rundir/demo_configs/test/config/demo-configs.data.xml tc-maker-1
oks_enable -d -s simple-demo-system ./rundir/demo_configs/test/config/demo-configs.data.xml tp-stream-writer

sed -i 's,<ref class="Variable" id="daqapp-cli-configuration"/>,<ref class="Variable" id="daqapp-cli-configuration"/>\n  <ref class="Variable" id="session-env-trace_file"/>,' rundir/demo_configs/test/config/demo-configs.data.xml 
sed -i "s,</oks-data>,<obj class=\"Variable\" id=\"session-env-trace_file\">\n <attr name=\"name\" type=\"string\" val=\"TRACE_FILE\"/>\n <attr name=\"value\" type=\"string\" val=\"${PWD}/log/${USER}_dunedaq.trace\"/>\n</obj>\n\n</oks-data>," rundir/demo_configs/test/config/demo-configs.data.xml 

cd rundir

killall drunc-controller 2>/dev/null
sleep 5
drunc-unified-shell ssh-standalone test/config/demo-configs.data.xml simple-demo-system boot wait 8 conf wait 3 start 99 enable-triggers wait 10 disable-triggers drain-dataflow stop-trigger-sources stop scrap terminate

tonM -n DFOModule DEBUG+21
tonM -n DFOModule DEBUG+22
tonM -n DFOModule DEBUG+23
tonM -n DFOModule DEBUG+24
tonM -n TRBModule DEBUG+21
tonM -n TRBModule DEBUG+22

killall drunc-controller 2>/dev/null
sleep 5
drunc-unified-shell ssh-standalone test/config/demo-configs.data.xml simple-demo-system boot wait 8 conf wait 3 start 101 enable-triggers wait 10 disable-triggers drain-dataflow stop-trigger-sources stop scrap terminate

sed -i 's,<attr name="request_timeout" type="u32" val="1000"/>,<attr name="request_timeout" type="u32" val="10"/>,' demo_configs/test/config/hsi-segment.data.xml
sed -i 's,<attr name="request_timeout" type="u32" val="10000"/>,<attr name="request_timeout" type="u32" val="10"/>,' demo_configs/test/config/trigger-segment.data.xml

killall drunc-controller 2>/dev/null
sleep 5
drunc-unified-shell ssh-standalone test/config/demo-configs.data.xml simple-demo-system boot wait 8 conf wait 3 start 201 enable-triggers wait 10 disable-triggers drain-dataflow stop-trigger-sources stop scrap terminate

sed -i 's,<attr name="request_timeout" type="u32" val="10"/>,<attr name="request_timeout" type="u32" val="0"/>,' demo_configs/test/config/hsi-segment.data.xml
sed -i 's,<attr name="request_timeout" type="u32" val="10"/>,<attr name="request_timeout" type="u32" val="0"/>,' demo_configs/test/config/trigger-segment.data.xml

killall drunc-controller 2>/dev/null
sleep 5
drunc-unified-shell ssh-standalone test/config/demo-configs.data.xml simple-demo-system boot wait 8 conf wait 3 start 301 enable-triggers wait 10 disable-triggers drain-dataflow stop-trigger-sources stop scrap terminate

killall drunc-controller 2>/dev/null

tshowt | egrep 'PAUSE|RESUME|TRBModule' | egrep 'PAUSE|RESUME|Received' | egrep 'PAUSE|RESUME|6\.0|7\.0'

[bieryAtFnal]

Here is sample TRACE output from the last line in the instructions listed above. Recall that the tshow command, by default, shows the TRACE messages in reverse time order.

In the sample output...

• in run 101, we see the fragments from the HSI and Trigger in TR 6 arriving ~1 second later than the fragments from the readout app. In run 101, the request_timeout values for the HSI and Trigger apps were set to 1 sec or larger.
• in run 201, we see the fragments from the HSI and Trigger in TR 6 arriving ~1 second later than the fragments from the readout app. In run 201, the request_timeout values for the HSI and Trigger apps were set to 10 msec.
• in run 301, we see the fragments from the HSI and Trigger in TR 6 arriving at essentially the same time as the fragments from the readout app. In run 301, the request_timeout values for the HSI and Trigger apps were set to zero.

160 09-18 14:50:05.268882 2362446 2369109  90                                 MLTModule:179 LOG do_pause: ******* Triggers PAUSED! in run 301 *********
 345 09-18 14:50:00.989687 2362347 2367996  17                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 106
 346 09-18 14:50:00.989671 2362347 2367996  17                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 102
 347 09-18 14:50:00.989655 2362347 2367996  17                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 104
 348 09-18 14:50:00.989639 2362347 2367996  17                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 101
 349 09-18 14:50:00.989611 2362347 2367996  17                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 105
 350 09-18 14:50:00.989594 2362347 2367996  17                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 103
 351 09-18 14:50:00.989579 2362347 2367996  17                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 100
 352 09-18 14:50:00.989556 2362347 2367996  17                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 107
 353 09-18 14:50:00.989532 2362347 2367996  17                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: HW_Signals_Interface id: 1
 354 09-18 14:50:00.989523 2362347 2367996  17                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Trigger id: 10000
 378 09-18 14:49:59.989989 2362388 2367991  18                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 106
 379 09-18 14:49:59.989971 2362388 2367991  18                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 103
 380 09-18 14:49:59.989954 2362388 2367991  18                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 104
 381 09-18 14:49:59.989936 2362388 2367991  18                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 101
 382 09-18 14:49:59.989920 2362388 2367991  18                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 107
 383 09-18 14:49:59.989902 2362388 2367991  18                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 100
 384 09-18 14:49:59.989880 2362388 2367991  18                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 102
 385 09-18 14:49:59.989854 2362388 2367991  18                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: HW_Signals_Interface id: 1
 386 09-18 14:49:59.989849 2362388 2367991  18                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 105
 387 09-18 14:49:59.989799 2362388 2367991  18                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Trigger id: 10000
 503 09-18 14:49:57.024378 2362301 2367997  18                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 2.0 from subsystem: Detector_Readout id: 100
 504 09-18 14:49:57.024369 2362301 2367997  18                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 2.0 from subsystem: Detector_Readout id: 102
 505 09-18 14:49:57.024361 2362301 2367997  18                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 2.0 from subsystem: Detector_Readout id: 103
 506 09-18 14:49:57.024340 2362301 2367997  18                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 2.0 from subsystem: Detector_Readout id: 101
 510 09-18 14:49:57.021125 2362347 2367996  81                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 1.0 from subsystem: Detector_Readout id: 106
 511 09-18 14:49:57.021089 2362347 2367996  81                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 1.0 from subsystem: Detector_Readout id: 100
 512 09-18 14:49:57.021064 2362347 2367996  81                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 1.0 from subsystem: Detector_Readout id: 105
 513 09-18 14:49:57.021043 2362347 2367996  81                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 1.0 from subsystem: Detector_Readout id: 102
 514 09-18 14:49:57.021020 2362347 2367996  81                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 1.0 from subsystem: Detector_Readout id: 107
 515 09-18 14:49:57.020989 2362347 2367996  81                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 1.0 from subsystem: Detector_Readout id: 101
 516 09-18 14:49:57.020966 2362347 2367996  81                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 1.0 from subsystem: Detector_Readout id: 103
 517 09-18 14:49:57.020873 2362347 2367996  81                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 1.0 from subsystem: Detector_Readout id: 104
 518 09-18 14:49:57.020780 2362347 2367996  81                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 1.0 from subsystem: HW_Signals_Interface id: 1
 519 09-18 14:49:57.020750 2362347 2367996  81                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 1.0 from subsystem: Trigger id: 10000
 598 09-18 14:49:56.013094 2362388 2367991  82                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 0.0 from subsystem: Detector_Readout id: 101
 599 09-18 14:49:56.013074 2362388 2367991  82                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 0.0 from subsystem: Detector_Readout id: 107
 600 09-18 14:49:56.013052 2362388 2367991  82                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 0.0 from subsystem: Detector_Readout id: 103
 601 09-18 14:49:56.012988 2362388 2367991  82                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 0.0 from subsystem: Detector_Readout id: 105
 602 09-18 14:49:56.012969 2362388 2367991  82                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 0.0 from subsystem: Detector_Readout id: 102
 603 09-18 14:49:56.012950 2362388 2367991  82                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 0.0 from subsystem: Detector_Readout id: 106
 604 09-18 14:49:56.012930 2362388 2367991  82                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 0.0 from subsystem: Detector_Readout id: 104
 605 09-18 14:49:56.012905 2362388 2367991  82                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 0.0 from subsystem: HW_Signals_Interface id: 1
 606 09-18 14:49:56.012895 2362388 2367991  82                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 0.0 from subsystem: Detector_Readout id: 100
 607 09-18 14:49:56.012793 2362388 2367991  82                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 0.0 from subsystem: Trigger id: 10000
 695 09-18 14:49:55.180449 2362446 2368337  18                                 MLTModule:191 LOG do_resume: ******* Triggers RESUMED! in run 301 *********
3060 09-18 14:49:16.083512 2353702 2359120 104                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 14.0 from subsystem: Trigger id: 10000
3061 09-18 14:49:16.083500 2353702 2359120 104                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 14.0 from subsystem: HW_Signals_Interface id: 1
3089 09-18 14:49:16.019858 2353783 2359747  10                                 MLTModule:179 LOG do_pause: ******* Triggers PAUSED! in run 201 *********
3095 09-18 14:49:16.011286 2353748 2359140 104                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 13.0 from subsystem: HW_Signals_Interface id: 1
4207 09-18 14:49:12.011221 2353748 2359140 104                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: HW_Signals_Interface id: 1
4498 09-18 14:49:11.111974 2356926 2359133  41                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Trigger id: 10000
4499 09-18 14:49:11.111958 2356926 2359133  41                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: HW_Signals_Interface id: 1
4500 09-18 14:49:11.108677 2353748 2359140 104                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 100
4501 09-18 14:49:11.108623 2353748 2359140 104                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 105
4502 09-18 14:49:11.108566 2353748 2359140 104                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 103
4503 09-18 14:49:11.108512 2353748 2359140 104                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 106
4504 09-18 14:49:11.108455 2353748 2359140 104                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 101
4505 09-18 14:49:11.108395 2353748 2359140 104                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 104
4506 09-18 14:49:11.108343 2353748 2359140 104                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Trigger id: 10000
4507 09-18 14:49:11.108337 2353748 2359140 104                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 102
4508 09-18 14:49:11.108270 2353748 2359140 104                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 107
4737 09-18 14:49:10.108235 2356926 2359133  41                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 104
4738 09-18 14:49:10.108156 2356926 2359133  41                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 103
4739 09-18 14:49:10.108081 2356926 2359133  41                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 106
4740 09-18 14:49:10.108011 2356926 2359133  41                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 101
4741 09-18 14:49:10.107931 2356926 2359133  41                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 105
4742 09-18 14:49:10.107860 2356926 2359133  41                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 102
4743 09-18 14:49:10.107797 2356926 2359133  41                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 107
4744 09-18 14:49:10.107713 2356926 2359133  41                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 100
5665 09-18 14:49:07.098755 2356926 2359133 105                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 0.0 from subsystem: Trigger id: 10000
5898 09-18 14:49:05.863828 2353783 2359414 121                                 MLTModule:191 LOG do_resume: ******* Triggers RESUMED! in run 201 *********
6860 09-18 14:48:28.871945 2334065 2339308  14                                 MLTModule:179 LOG do_pause: ******* Triggers PAUSED! in run 101 *********
6963 09-18 14:48:26.047043 2333988 2338622  66                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 11.0 from subsystem: Detector_Readout id: 104
6964 09-18 14:48:26.047038 2333988 2338622  66                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 11.0 from subsystem: Detector_Readout id: 102
6965 09-18 14:48:26.047033 2333988 2338622  66                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 11.0 from subsystem: Detector_Readout id: 106
6966 09-18 14:48:26.046979 2333988 2338622  66                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 11.0 from subsystem: Detector_Readout id: 101
6967 09-18 14:48:26.046974 2333988 2338622  66                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 11.0 from subsystem: Detector_Readout id: 105
6968 09-18 14:48:26.046968 2333988 2338622  66                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 11.0 from subsystem: Detector_Readout id: 100
6969 09-18 14:48:26.046962 2333988 2338622  66                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 11.0 from subsystem: Detector_Readout id: 107
6970 09-18 14:48:26.046948 2333988 2338622  66                                 TRBModule:388 D22 read_fragments: df-01-trb Received fragment for trigger/sequence_number 11.0 from subsystem: Detector_Readout id: 103
6971 09-18 14:48:26.033177 2334016 2338649 102                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 10.0 from subsystem: Trigger id: 10000
7027 09-18 14:48:24.835283 2334016 2338649 102                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: HW_Signals_Interface id: 1
7052 09-18 14:48:24.032547 2334016 2338649 100                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Trigger id: 10000
7070 09-18 14:48:23.935603 2337039 2338626 100                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: HW_Signals_Interface id: 1
7071 09-18 14:48:23.932221 2334016 2338649 100                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 100
7072 09-18 14:48:23.932147 2334016 2338649 100                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 107
7073 09-18 14:48:23.932088 2334016 2338649 100                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 103
7074 09-18 14:48:23.932037 2334016 2338649 100                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 104
7075 09-18 14:48:23.931982 2334016 2338649 100                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 102
7076 09-18 14:48:23.931909 2334016 2338649 100                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 106
7077 09-18 14:48:23.931842 2334016 2338649 100                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 101
7078 09-18 14:48:23.931757 2334016 2338649 100                                 TRBModule:388 D22 read_fragments: df-03-trb Received fragment for trigger/sequence_number 7.0 from subsystem: Detector_Readout id: 105
7079 09-18 14:48:23.835244 2337039 2338626 100                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Trigger id: 10000
7096 09-18 14:48:22.932696 2337039 2338626  39                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 101
7097 09-18 14:48:22.932639 2337039 2338626  39                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 106
7098 09-18 14:48:22.932592 2337039 2338626  39                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 100
7099 09-18 14:48:22.932540 2337039 2338626  39                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 104
7100 09-18 14:48:22.932490 2337039 2338626  39                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 103
7101 09-18 14:48:22.932443 2337039 2338626  39                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 105
7102 09-18 14:48:22.932403 2337039 2338626  39                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 102
7103 09-18 14:48:22.932309 2337039 2338626  39                                 TRBModule:388 D22 read_fragments: df-02-trb Received fragment for trigger/sequence_number 6.0 from subsystem: Detector_Readout id: 107
7416 09-18 14:48:18.724096 2334065 2338904  84                                 MLTModule:191 LOG do_resume: ******* Triggers RESUMED! in run 101 *********
8370 09-18 14:47:40.719394 2324668 2329494  83                                 MLTModule:179 LOG do_pause: ******* Triggers PAUSED! in run 99 *********
8539 09-18 14:47:30.628878 2324668 2329260   3                                 MLTModule:191 LOG do_resume: ******* Triggers RESUMED! in run 99 *********