Frame metadata timestamps

[Folyz]

Required Info
Camera Model	D435
Firmware Version	05.09.14.00
Operating System & Version	Win 10
Platform	PC
SDK Version	2.11.1

Question about timestamps

I have a question about how the timestamps that are present in the metadata are generated. I'm using Nuitrack to extract skeletons from the image data and I need to synchronize this data with other devices. My main question is: to what event do the timestamps belong and what is the clock from which they are generated? My guess so far is:

• SENSOR_TIMESTAMP: Device clock / sensor starts taking the image
• FRAME_TIMESTAMP: Device clock / frame starts to transfer to the driver
• BACKEND_TIMESTAMP: PC clock / frame starts to transfer to the driver
• TIME_OF_ARRIVAL: PC clock / frame is transfered to the driver

Hopefully you can give me more clarity on this, I couldn't find it anywhere.

[ev-mp]

Hello @Folyz , Indeed the documentation for the metadata origin should be improved, nevertheless with regards to depth stream: Sensor(optical)_timestamp is a Firmware-generated timestamp that marks middle of sensor exposure. Frame_timestamp - Generated in FW; designates the beginning of UVC frame transmission. (the first USB chunk sent towards host). Sensor_timestamp and Frame_timestamp share the same clock and require the kernel patch for Video4Linux and registry patch for Windows OS.

Backend_timestamp - Host clock applied to kernel-space buffers (URB) when the data from USB controller is copied to OS kernel. (HW->Kernel transition) Time_of_arrival - Host clock for the time when the frame buffer reaches Librealsense (kernel->user space transition).

The relations hold: Sensor_timestamp << Frame_timestamp. Backend_timestamp << Time_of_arrival.

The clocks of the first pair of timestamps and the second pairs are not related.

[Folyz]

Thanks for your answer, this is very clarifying. I also assume the best way to synchronize frames to the clock is currently using the frame_timestamp and backend_timestamp to translate the FW clock to the host clock? Or is there any other possibility for synchronization of these clocks?

[bentank]

Is there a way to query the hardware clock of the Realsense module that is used to generate both Sensor_timestamp and Frame_timestamp?

I want the ability to correlate the sensor timestamp to my host system timestamp and receiving the this per frame does not give me a reference point.

[Folyz]

I have the same question as bentank, currently there seems to be a time delay between sending the data and receiving thus backend_timestamp - frame_timestamp is inaccurate.

[ev-mp]

Unfortunately there is no reliable way to translate HW timestamps to host time without making assumptions regarding USB layer. One can empirically establish the upper bound for Device->Host transfer time by analyzing USB traffic with external tools/usb sniffer. Some users who required high-precision sync, also tried to sync clocks by taking pictures of msec-resolution images of clocks and then comparing on-screen image time with the FW-generated timestamp. A similar method is also employed in one of librealsense demos that profiles end-to-end frame propagation (latency).

Such observations, however, would probably hold to a single well-established HW setup and cannot be extrapolated.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] @Folyz is ev-mp' s reply can solve your issue

[bentank]

Can we please add the functionality to request current sensor timestamp from Realsense firmware? This would mean ability to arbitrarily sample the sensor timestamp from the host. This would allow for correlation of sensor timestamp to a different timebase from the host and should be nearly trivial to implement in FW.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Hi @bentank,

You can enable "ENFORCE_METADATA" CMAKE option to get the sensor timestamp. Let us know how it works for you.

[bentank]

Please see my comments above. While adding that flag helps to get the sensor timestamp, it does not help with correlating that timestamp to my host's timestamp.

Can we please add the functionality to request current sensor timestamp from Realsense firmware? This would mean ability to arbitrarily sample the sensor timestamp from the host. This would allow for correlation of sensor timestamp to a different timebase from the host and should be nearly trivial to implement in FW.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Hi @bentank,

I can see your point. Will check internally about this feature request.

[bentank]

@RealSense-Customer-Engineering

Is there any update on this request? The same API for getSensorTimeStamp() is requested here as well: https://github.com/IntelRealSense/librealsense/issues/2148#issuecomment-413456656

Thanks!

[bentank]

@dorodnic, do you have insight or update on this issue?

Thanks for all the support on the issues in this repo!

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment] Ticket being closed due to inactivity for 30+ days.

[bentank]

@RealSense-Customer-Engineering - the ball is in your court on this issue. Why are you closing it? Do you know want to implement it?

Please add some insight here?

[FHainzl]

Very 'interesting' move to promise to "check internally about this feature request", and then close the issue due to inactivity...

[MaheshAbnave]

Does the global timestamp resolve this issue? That is if I make a call to sensor.set_option(RS2_OPTION_GLOBAL_TIME_ENABLED, 1), will it provide frame timestamp synced with host time?

[MartyG-RealSense]

Hi @MaheshAbnave The global timestamp generates a common timestamp for all streams by comparing device time and computer time, as described at https://github.com/IntelRealSense/librealsense/pull/3909

[monson-li]

Hello @Folyz , Indeed the documentation for the metadata origin should be improved, nevertheless with regards to depth stream: Sensor(optical)_timestamp is a Firmware-generated timestamp that marks middle of sensor exposure. Frame_timestamp - Generated in FW; designates the beginning of UVC frame transmission. (the first USB chunk sent towards host). Sensor_timestamp and Frame_timestamp share the same clock and require the kernel patch for Video4Linux and registry patch for Windows OS.

Backend_timestamp - Host clock applied to kernel-space buffers (URB) when the data from USB controller is copied to OS kernel. (HW->Kernel transition) Time_of_arrival - Host clock for the time when the frame buffer reaches Librealsense (kernel->user space transition).

The relations hold: Sensor_timestamp << Frame_timestamp. Backend_timestamp << Time_of_arrival.

The clocks of the first pair of timestamps and the second pairs are not related.

Hello @ev-mp , can you explain the concept of "time of arrival" more deeply. I guess that means data transfer from kernel to visualization？

[MartyG-RealSense]

Hi @monson-li The time_of_arrival timestamp is provided by the Host (EPOCH) clock and represents frame transfer from the USB driver to the librealsense SDK.

There is further explanation at https://github.com/IntelRealSense/librealsense/issues/6061#issuecomment-599205181

[monson-li]

Thanks for your reply.