Clock skew logging

[trey0]

The system monitor already watches for MLP/LLP clock skew and raises a fault if it exceeds a threshold. In addition to fault checking, it would be useful to log the clock skew (i.e. log near-simultaneous timestamps between a pair of processors, so the skew can be estimated later). This could help with debugging and also with post-activity data analysis, such as correcting the skew to get better results. Skew correction might be beneficial even if the skew did not exceed the fault threshold.

Some considerations:

• It seems logical to use the system monitor on the MLP to log the skew, since it's already receiving timestamped heartbeats and using that to check for the clock skew fault.
• We recently noticed large HLP/MLP clock skew during the ISAAC-2 activity, so skew logging makes sense for the HLP as well as the LLP.
• We probably don't need to log clock skew for every heartbeat from every module, since clock skew doesn't tend to change frequently. Perhaps one message every 10 seconds would suffice. Maybe specify a short list of modules to report clock skew for (say 1-2 each on HLP and LLP).
• The clock skew reporting could be via generic rosout message or it could be via a new dedicated ROS message to reduce bandwidth on the rosout channel and simplify implementing automated clock skew correction later (if needed).

[oleg-alexandrov]

Strongly suggest keeping track of the HLP and LLP clock difference. Out of sync clocks is causing big issues in processing ISAAC data.

[bcoltin]

Marina added image time to log for isaac. Katie will look into logging more info about the fault for clock skew. Also implement for HLP.

[trey0]

One approach for this is periodically logging simultaneous timestamps using SSH and the UNIX date command. PR for a draft script in NASA-internal repo at https://babelfish.arc.nasa.gov/bitbucket/projects/ASTROBEE/repos/astrobee_ops/pull-requests/3/overview