search

microsoft / HoloLens2ForCV

Sample code and documentation for using the Microsoft HoloLens 2 for Computer Vision research.
MIT License
489 stars 145 forks source link

VLC camera frame synchronization #98

Closed kovendijozsef closed 3 years ago

kovendijozsef commented 3 years ago

Hello,

I have been trying to create disparity maps from the 2 front facing VLC cameras. For this it is important that the images captured by these cameras are taken at the same time. After comparing the timestamps of the Left and Right front cameras I have found out that the images were not taken at the same time and in most cases the time difference between the image pairs are quite big thus the disparity map is wrong.

My question is: is there a way to capture images with the Left front and Right front cameras at the same time so synchronize them?

Thanks in advance!

palffybalazs commented 3 years ago

Same problem here. Is there a way to solve this?

maaron commented 3 years ago

I don't have a definitive answer, but something that may help. The timestamps for each frame have two fields, "SensorTicks", and "HostTicks". Although the samples in this repo use HostTicks exclusively, I've noticed that the SensorTicks field does tend to have exactly identical timestamps between LF and RF (and presumably RR/LL frames too, but I haven't checked). You'll also notice that the deltas between frames is much more consistent, corresponding to the 30fps frame rate. HostTicks, on the other hand, is much more variable. This leads me to believe that the SensorTicks field is from a clock that is driving the cameras, (and maybe everything running off the HPU?), whereas HostTicks is from the SoC and probably not sync'ed. Marc Pollefeys confirms in this video (https://www.youtube.com/watch?v=0Ov2PdlBKE0) that the cameras are synchronized in the hardware. Although the video is about HL1, I have to assume the same is true in HL2.

So in short, I think you can use ResearchModeSensorTimestamp.SensorTicks to correlate matched pairs.

kovendijozsef commented 3 years ago

Thank you maaron for your quick reply!

I managed to apply the changes you suggested. I also turned off some of the sensors that I did not need (e. g. PV camera). The data I managed to get this way is indeed much better but still not perfect ( I have not figured out yet why, but sometimes the LF and RF cameras tend to "desynchronize" for a few frames and by that I mean that the RF camera misses out 3 - 5 frames that the LF camera captures).

zhangyk18 commented 3 years ago

https://github.com/microsoft/HoloLensForCV/blob/master/Samples/ArUcoMarkerTracker/AppMain.cpp#L252 and 

auto absoluteTimestamp = m_converter.RelativeTicksToAbsoluteTicks(HundredsOfNanoseconds((long long)prevTimestamp)).count();

may be helpful to you.