add warning for late timesync messages

[strilov]

Using Florian's config, where the fake HSI emulator is running on np04-srv-025, and the readout apps running on np04-srv-028, I can indeed see TimeSync messages with system time around 90s behind current system time. This does seem to correspond to the different in date/time seen between np04-srv-025 and np04-srv-028, where 025 is ahead of 028.

2021-May-27 18:05:07,270 WARNING [void dunedaq::timinglibs::TimestampEstimator::estimator_thread_fn(std::unique_ptr<dunedaq::appfwk::DAQSource<dunedaq::dfmessages::TimeSync> >&) at /nfs/home/strilov/daq_250/hsi/sourcecode/timinglibs/src/TimestampEstimator.cpp:87] The most recent TimeSync message is behind current system time by 87594460 us.
2021-May-27 18:05:07,280 WARNING [void dunedaq::timinglibs::TimestampEstimator::estimator_thread_fn(std::unique_ptr<dunedaq::appfwk::DAQSource<dunedaq::dfmessages::TimeSync> >&) at /nfs/home/strilov/daq_250/hsi/sourcecode/timinglibs/src/TimestampEstimator.cpp:87] The most recent TimeSync message is behind current system time by 87604285 us.
2021-May-27 18:05:07,290 WARNING [void dunedaq::timinglibs::TimestampEstimator::estimator_thread_fn(std::unique_ptr<dunedaq::appfwk::DAQSource<dunedaq::dfmessages::TimeSync> >&) at /nfs/home/strilov/daq_250/hsi/sourcecode/timinglibs/src/TimestampEstimator.cpp:87] The most recent TimeSync message is behind current system time by 87614388 us.
2021-May-27 18:05:07,300 WARNING [void dunedaq::timinglibs::TimestampEstimator::estimator_thread_fn(std::unique_ptr<dunedaq::appfwk::DAQSource<dunedaq::dfmessages::TimeSync> >&) at /nfs/home/strilov/daq_250/hsi/sourcecode/timinglibs/src/TimestampEstimator.cpp:87] The most recent TimeSync message is behind current system time by 87624479 us.
2021-May-27 18:05:07,310 WARNING [void dunedaq::timinglibs::TimestampEstimator::estimator_thread_fn(std::unique_ptr<dunedaq::appfwk::DAQSource<dunedaq::dfmessages::TimeSync> >&) at /nfs/home/strilov/daq_250/hsi/sourcecode/timinglibs/src/TimestampEstimator.cpp:87] The most recent TimeSync message is behind current system time by 87634575 us.
2021-May-27 18:05:07,320 WARNING [void dunedaq::timinglibs::TimestampEstimator::estimator_thread_fn(std::unique_ptr<dunedaq::appfwk::DAQSource<dunedaq::dfmessages::TimeSync> >&) at /nfs/home/strilov/daq_250/hsi/sourcecode/timinglibs/src/TimestampEstimator.cpp:87] The most recent TimeSync message is behind current system time by 87644682 us.

Is there anything else we should do in the code?

Can we run NTP at NP04, especially for these types of tests?

[bieryAtFnal]

There are a lot of warning messages being produced by the new code (examples below).

Could it be that the test should be against 1000000 (one million microseconds)?

2021-May-27 11:42:47,227 WARNING [void dunedaq::timinglibs::TimestampEstimator::estimator_thread_fn(std::unique_ptr<dunedaq::appfwk::DAQSource<dunedaq::dfmessages::TimeSync> >&) at /home/biery/dunedaq/27May/sourcecode/timinglibs/src/TimestampEstimator.cpp:87] The most recent TimeSync message is behind current system time by 68745 us.
2021-May-27 11:42:47,237 WARNING [void dunedaq::timinglibs::TimestampEstimator::estimator_thread_fn(std::unique_ptr<dunedaq::appfwk::DAQSource<dunedaq::dfmessages::TimeSync> >&) at /home/biery/dunedaq/27May/sourcecode/timinglibs/src/TimestampEstimator.cpp:87] The most recent TimeSync message is behind current system time by 78863 us.
2021-May-27 11:42:47,247 WARNING [void dunedaq::timinglibs::TimestampEstimator::estimator_thread_fn(std::unique_ptr<dunedaq::appfwk::DAQSource<dunedaq::dfmessages::TimeSync> >&) at /home/biery/dunedaq/27May/sourcecode/timinglibs/src/TimestampEstimator.cpp:87] The most recent TimeSync message is behind current system time by 88990 us.

[strilov]

Thanks Kurt. Fix pushed.

[floriangroetschla]

Looks good. For future tests the clocks at np04 just have to be synchronized a bit better.