Frame timestamps

[ckemere]

We use CLOCK_MONOTONIC to synchronize between the miniscope data and our behavioral data acquisition code. I believe you've chosen to use system_clock rather than steady_clock (which I think would give us the version we use). Is there a reason?

[ximion]

By default, PoMiDAQ uses C++11's std::chrono::steady_clock, which on Linux is backed by CLOCK_MONOTONIC. The system clock isn't used for timings, it is only consulted to get the OSes time since epoch (= UNIX timestamp in most cases).

[ckemere]

To clarify, we use the raw CLOCK_MONOTONIC values to synchronize data. It appears that if you click the "Use Unix Timestamps" button, you get CLOCK_REALTIME values (boost system_clock), but if you don't, the timestamps start at zero. Is there a way to get just the raw CLOCK_MONOTONIC values?

I wrote a quick test function and verified that the timestamps we find in the timestamps.csv file are the "system clock" (This is in a system whose uptime is at 157 days, so the values are quite divergent.):

include <boost/chrono.hpp>
#include <iostream>

using namespace boost::chrono;

int main()
{
  std::cout << "system " << system_clock::now() << '\n';
#ifdef BOOST_CHRONO_HAS_CLOCK_STEADY
  std::cout << "steady " << steady_clock::now() << '\n';
#endif
}

[ckemere]

Would you accept a pull request that replaced the "0" at the top of the timestamp file with the actual value of steady_clock() for the first frame? It would be helpful to us. (It also aligns with the miniscope project's approach, I think, so it might be helpful to others.)

[ximion]

So, the reason why CLOCK_MONOTONIC isn't used as starting point when "Use UNIX timestamps" is selected is that this clock starts at an unspecified point in time - may be the system startup time, may be the program start time, but it's an unspecified timepoint which makes it a bad idea for synchronizing things on the same machine (Linux uses the system start time as base, but that isn't a guaranteed feature at all any may change and still be valid) and makes it terrible for synchronizing data between machines (no alignment possible).

Changing the zero-based start would mess with all API consumers, especially Syntalos which is the primary external consumer of this API for time alignment between multiple data sources. So changing the is a definite no. The UNIX timestamp option however exists pretty much for your usecase, which, as I recall, is aligning the time recorded by PoMiDAQ with that of an external program. For that, I would be less strict but I do think using steady_clock for any alignment between processes is wrong. The steady_clock should only be used for measuring intervals, using it between processes is by the API description not a great idea (POSIX, C++, Linux and Windows API agree on this, for once ^^). If PoMiDAQ switched to using it, the times recorded on different computers would also stop to be comparable, which might be a problem for some users (at the moment the times are at least comparable if the computers run the same OS). What do you think? Would it be possible to use the UNIX epoch as base for the other tool / is there a reason why steady_clock is used for between-app time sync in the other tool, except for timestamp_epoch_at_start + steady_time_since_start?

[ckemere]

We use these timestamps exclusively for synchronizing within a machine. I guess I can imagine trying to do high-resolution NTP in order to get the CLOCK_REALTIME correct and then using it across machines., But my major concern with that would be that a random automatic process (i.e., NTP) or the user messing around would cause the system time to change during recording. That would potentially make the timestamps non-monotonic. Non-motonic timestamps are obviously much less trivial to align than monotonic ones.

Is there a place where timestamp_epoch_at_start is recorded in the file? I still don't understand how the timestamps can be used for synchronization in the current version if they start at 0.

Inspired by your comment, I was reading up on the time standards. I guess "steady_clock" in the CPP standard is both monotonic and stable increment, the later of which CLOCK_MONOTONIC doesn't guarantee. AFAIK, CLOCK_MONOTONIC (or CLOCK_MONOTONIC_RAW) is a Linux thing that is unlikely to change from being system-wide.

Regardless - I appreciate the pomidaq program and don't want to cause trouble! Thanks for all your speedy replies!

[ckemere]

Reading through the code, I think I misunderstood. It looks like perhaps for the "Use Unix Timestamps" setting what happens is that the first timestamp is the CLOCK_REALTIME time for the first frame, but then the rest are all the CLOCK_MONOTONIC difference from that point. Do I understand that correctly? That's much easier to understand....

[ximion]

It looks like perhaps for the "Use Unix Timestamps" setting what happens is that the first timestamp is the CLOCK_REALTIME time for the first frame, but then the rest are all the CLOCK_MONOTONIC difference from that point. Do I understand that correctly?

Yes, that's right :-) - Sorry, maybe I should have started with that as an explanation. PoMiDAQs internal times are of course monotonic (and in fact, always use the steady clock), but the initial timepoint zero is obtained from the system clock if the "UNIX timestamp" option is selected, or is set to an API-defined timepoint (zero by default). To make things even more complicated, we actually even use the frame timestamp the driver gives us, if that one is good enough. That is (usually) the time when the driver has received a frame, even before sending it to userspace, and is surprisingly precise for Miniscopes. If we can't use the driver time, a steady clock is used (whatever the OS prefers via the C++ API, CLOCK_MONOTONIC on Linux and whatever QueryPerformanceCounter returns on Windows). On Linux, we usually have a driver time from the uvcvideo driver.

The Syntalos tool can use zero-based times for synchronization, because its internal clock is shared between all DAQ modules (or modules are forced to align to the master clock). For separate tools this will of course not work, you can never start all at exactly the right time.

Btw, I actually analyzed the timestamps Syntalos generates (which are generated using the same code PoMiDAQ has) and the timestamps are very accurate with no clock drift even if you record 16h straight. The primary source of noise comes from the Miniscope framerate fluctuating wildly between 28 and 30.5 fps (not sure why that happens, at the moment I blame the DAQ hardware :-P).

Clocks on Linux are actually pretty involved, and there is lots of them. If you're interested, https://man7.org/linux/man-pages/man2/clock_getres.2.html has an excellent summary. I actually experimented with CLOCK_MONOTONIC_RAW for Syntalos, and think ultimately for what we are doing CLOCK_MONOTONIC is better. CLOCK_MONOTONIC_RAW will actually vary a lot (well... still in nanosecond range) between machines (and to my surprise occasionally also between boot cycles, not sure why) and simply represents a local oscillator, usually the CPU TSC. Those really aren't perfect, and for our timers we want a microsecond to be as close to the real deal as possible. The normal monotonic clock provides this adjusted version, so results are less odd if your local oscillator is bad. Any adjustments made to the monotonic clock frequency during a run haven't been measurable by me, quite likely I wasn't testing with a clock that had a resolution high enough to notice (µs might have been to coarse, and ns might have been better). So, the regular monotonic clock it is :-) Not sure what the raw variant is actually used for, but I can imagine it being useful for some hard-realtime applications as well as for synchronizing the time in a cluster of computers where only one gets the NTP signal and extremely tight synchronization is required.

[ckemere]

Perfect.

Yeah - we have a hardware clock on our data acquisition board to try to improve timing reliability, but I've convinced myself that millisecond precision is probably good enough for almost everything we'd want to do. I looked briefly at Syntalos - will you present it at SFN this year?

[ximion]

It's always a question of what data you acquire - for behavior, there's not much a mouse can do in 1ms (whisking activity possibly, but for that you'll have a different camera with higher temporal resolution anyway), but if the Miniscope's maximum framerate is 30fps, millisecond resolution is absolutely fine :-)

I would love to present it at SfN this year, but international travel is still difficult, even if you have a vaccine combination that the U.S. accepts to let you in. So we'll likely not go this year, but there's a good chance for next year! (and by then maybe there will be multiple projects completed with Syntalos (lots in progress at the moment, nothing done - behavior experiments are frustrating sometimes))