Changes in readout configuration, integration with DQM

[jmcarcell]

There are two major changes:

• DQM is included in the readout configuration for now
• Changes in the readout configuration using the publish-subscribe model. It goes together with some changes in readout that have been merged today

There is a new flag, --enable-dqm to enable DQM (by default disabled). The tests I have done with and without DQM with one or two readout apps have worked fine

[bieryAtFnal]

A couple of minor things that I noticed when I first started running some tests of these changes...

• the jcarcell/DQM-integration branch needed to be updated from the develop branch. I did the merge and will push it to the central repo in a bit. Juan, please check that this merge didn't cause any problems.
• it wasn't obvious to me that I needed to include the dqm repo in my software area, but there was a confgen error before I did that. So, I added it. Presumably, this won't be necessary once tomorrow's (Tuesday's) Nightly build becomes available.

[jmcarcell]

Whoops, sorry forgot about those changes. I had a look at the code and it doesn't interfere with our changes. I have done some testing and everything seems to be fine. Yeah, there were a few changes in the repo yesterday that went together with those changes. Actually today there is one more change to the DQM configuration so it will be necessary to build from develop.

[bieryAtFnal]

In general, the testing that I have done so far looks good.

However, when I test multiple runs with different run numbers in the same DAQ session, and I have DQM enabled, I see messages like the following in the RU logfile:

log_ruemu0_3335.txt:2021-Jul-28 15:50:49,199 ERROR [void dunedaq::dfmodules::TriggerRecordBuilder::do_work(std::atomic<bool>&) at /home/biery/dunedaq/28Jul_N0728/sourcecode/dfmodules/plugins/TriggerRecordBuilder.cpp:289] Unexpected Trigger Decisions: 51/101 while in run 102

Maybe the TDs aren't being flushed out of the DQM-related TRBuilder instance between runs?

Here are the commands that I used to demonstrate this issue:

python -m minidaqapp.nanorc.mdapp_multiru_gen --host-ru localhost -d /scratch/biery/data/frames.bin -o . -s 10 --enable-dqm mdapp_4proc_withDQM

nanorc mdapp_4proc_withDQM boot init conf start 100 resume wait 20 pause wait 3 stop wait 2 start 101 resume wait 20 pause wait 3 stop wait 1 start 102 resume wait 20 pause wait 3 stop wait 2 start 103 resume wait 20 pause wait 3 stop wait 1 scrap terminate

[jmcarcell]

So the current status before merging:

• When DQM is enabled and multiple readout units are running there will be errors for all readout units but the first about Unexpected Trigger Decisions. I'm working on that
• When DQM is enabled, the event building time for dataflow increases with time. Florian is working on that