DUNE-DAQ / timinglibs

0 stars 1 forks source link

Fix the HSI read errors at run start at ICEBERG when using external signal input #47

Closed bieryAtFnal closed 2 years ago

bieryAtFnal commented 2 years ago

In testing of dunedaq-v2.10.0 at ICEBERG with real HSI/TLU electronics, I've noticed messages like the following in the HSI logfile:

2022-Mar-15 12:04:26,459 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /home/nfs/dunedaq/daqsw/15Mar2.10.0/sourcecode/timinglibs/plugins/HSIReadout.cpp:226] Failed to read HSI events.
    was caused by: 2022-Mar-15 12:04:26,459 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /home/nfs/dunedaq/daqsw/15Mar2.10.0/sourcecode/timinglibs/plugins/HSIReadout.cpp:160] HSI endpoint not ready. Current state: 0x7

These errors seem to happen at the beginning of a run and then stop.

If these are a normal part of the startup of each run, maybe we can reduce their severity somehow? Of course, if they happen during a run, I expect that we will still want them to be errors.

glehmannmiotto commented 2 years ago

No run should start with endpoints not in running (0x8) state. The timestamp may be completely off. So I think that the severity is appropriate.

Giovanna

———————- CERN EP/DT-DI

On 18 Mar 2022, at 21:43, bieryAtFnal @.***> wrote:



In testing of dunedaq-v2.10.0 at ICEBERG with real HSI/TLU electronics, I've noticed messages like the following in the HSI logfile:

2022-Mar-15 12:04:26,459 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic&) at /home/nfs/dunedaq/daqsw/15Mar2.10.0/sourcecode/timinglibs/plugins/HSIReadout.cpp:226] Failed to read HSI events. was caused by: 2022-Mar-15 12:04:26,459 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic&) at /home/nfs/dunedaq/daqsw/15Mar2.10.0/sourcecode/timinglibs/plugins/HSIReadout.cpp:160] HSI endpoint not ready. Current state: 0x7

These errors seem to happen at the beginning of a run and then stop.

If these are a normal part of the startup of each run, maybe we can reduce their severity somehow? Of course, if they happen during a run, I expect that we will still want them to be errors.

— Reply to this email directly, view it on GitHubhttps://github.com/DUNE-DAQ/timinglibs/issues/47, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD6SRUYVIXVP43EMAYOM4BTVATTGFANCNFSM5RC3VTHA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: @.***>

bieryAtFnal commented 2 years ago

Good to know, thanks.

I'll be glad to change the content of this Issue to something more along the lines of "fix whatever underlying problem is causing these error messages to be produced at the start of every run at ICEBERG that uses the external signal input to the HSI/TLU electronics."

strilov commented 2 years ago

Hi. There is a pathology here where the period between the issuing of the conf command from nanorc and the HSI endpoint reaching state 0x8 (ready) is longer than the period between the issuing of the conf command and the issuing of the start command.

I will have a look at the best way to prevent this from happening.

strilov commented 2 years ago

Hi @bieryAtFnal, I've made a PR here and in daqconf (https://github.com/DUNE-DAQ/daqconf/pull/46) which should solve this issue. Are you able to test this for merging?

bieryAtFnal commented 2 years ago

Yes, I will run some tests this afternoon.

bieryAtFnal commented 2 years ago

I wanted to use a recently nightly build (e.g. N22-05-11) as the basis for my tests, but it seems that the changes on the feature/interpert_hw_info branch in the daqconf repo are significantly behind the state of the code on the develop branch of daqconf. I tried to re-merge from develop into feature/interpert_hw_info, but there were conflicts that I didn't know how to resolve. So, @strilov , could you do that merge? Thanks

strilov commented 2 years ago

Hi @bieryAtFnal, that branch is behind; https://github.com/DUNE-DAQ/daqconf/tree/feature/hw_state_check_iomanager is the one to use.

bieryAtFnal commented 2 years ago

Sorry for that mistake. I've switched to using the correct branch.

However, I still see error messages.

To double-check that I'm still not doing something wrong, I will copy the steps that I use below. And, below that, I will copy the error messages that I see. If I'm still doing something wrong, please let me know.

# in one (clean/fresh) shell window
cd <workdir>
source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest-spack
dbt-create -c -n N22-05-11 11MayHSITest
cd 11MayHSITest
cd sourcecode
git clone https://github.com/DUNE-DAQ/daqconf.git -b feature/hw_state_check_iomanager
git clone https://github.com/DUNE-DAQ/timinglibs.git -b feature/feature/interpert_hw_info
cd ..
dbt-workarea-env
dbt-build.py -j 16
dbt-workarea-env
daqconf_timing_gen --host-thi iceberg01-priv --host-tmc iceberg01-priv --master-device-name BOREAS_TLU_ICEBERG --clock-speed-hz 62500000 timing_partition_config
nanotimingrc timing_partition_config boot init conf wait 1200 scrap terminate

# in another (clean/fresh) shell window
cd <workdir>/11MayHSITest
source ./dbt-env.sh
dbt-workarea-env
daqconf_multiru_gen --use-hsi-hw --host-hsi iceberg01-priv --control-hsi-hw --hsi-device-name BOREAS_TLU_ICEBERG --hsi-source 1 --ttcm-s1 1 --hsi-re-mask 1 --host-timing iceberg01-priv --control-timing-partition --host-tprtc iceberg01-priv --timing-partition-master-device-name BOREAS_TLU_ICEBERG -t 1.0 --hsi-trigger-type-passthrough --use-fake-data-producers --clock-speed-hz 62500000 mdapp_fdp_hsipulser
tmprun=111; runduration=30; waitAfterStop=2; nanorc mdapp_fdp_hsipulser/ boot init conf start ${tmprun} wait ${runduration} stop wait ${waitAfterStop} scrap terminate
egrep 'ERROR|WARNING' log_*
log_hsi_3333.txt:2022-May-11 14:54:39,646 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,646 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,648 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,648 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,649 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,649 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,651 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,651 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,652 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,652 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,653 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,653 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,655 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,655 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,656 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,656 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,657 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,657 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,659 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,659 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,660 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,660 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,661 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,661 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,662 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,662 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,664 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,664 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,665 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,665 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,666 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,666 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,667 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,667 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,669 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,669 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,670 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,670 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,671 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,671 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,672 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,672 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,674 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,674 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,675 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,675 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,676 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,676 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,677 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,677 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,679 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,679 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,680 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,680 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,681 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,681 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,682 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,682 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,683 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events.
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,683 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,697 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events. -- 10 similar messages suppressed, last occurrence was at 2022-May-11 14:54:39,696243
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,697 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
log_hsi_3333.txt:2022-May-11 14:54:39,824 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:239] Failed to read HSI events. -- 100 similar messages suppressed, last occurrence was at 2022-May-11 14:54:39,823459
log_hsi_3333.txt:   was caused by: 2022-May-11 14:54:39,824 ERROR [virtual void dunedaq::timinglibs::HSIReadout::do_hsievent_work(std::atomic<bool>&) at /tmp/root/spack-stage/spack-stage-timinglibs-N22-05-11-s2dkxinte4mhhelzflr3znpfecaxg5ns/spack-src/plugins/HSIReadout.cpp:158] HSI endpoint not ready. Current state: 0x7
strilov commented 2 years ago

Hi @bieryAtFnal, I notice that there is an extra feature/ in the timinglibs clone command above. Also, dbt-build.py needs to be just dbt-build. I think this means that both timinglibs and daqconf still come from the nightly.

bieryAtFnal commented 2 years ago

Wow, sorry for all the dumb mistakes, thanks for catching them. Tests with the changes look good to me, and I'll add approvals to the PRs.

strilov commented 2 years ago

Thanks. Now merged.