search

rpng / open_vins

An open source platform for visual-inertial navigation research.
https://docs.openvins.com
GNU General Public License v3.0
2.22k stars 652 forks source link

Theoretical question regarding global vs rolling shutter behaviour and accounting for it with ZED2 and D455 #464

Open Aishkrish18 opened 3 months ago

Aishkrish18 commented 3 months ago

I have a question. Couple of months ago, I had performed detailed tests between ZED2 stereo inertial mode and D455 with OpenVINS , resulting in good performance with ZED2 camera. But from some of the issues, i see that it is a rolling shutter camera, and the algorithm doesn't support rolling shutter. But D455 on the other hand gave relatively poor performance with bigger offset compared to ZED2 although it is a global shutter camera. This is with ZED2 zed2_openvins drawio

This is with D455 openvins_d455 drawio Both of them are compared in reference to motion capture system. May i get some insights on what could be the reason for this behaviour?

Aishkrish18 commented 3 months ago

to put it better, how does zed2 perform so well although its a rolling shutter camera, I'm unable to reason out this part

Userpc1010 commented 3 months ago

Why are you sure that the camera is the reason? The VINS system is primarily an inertial navigation system corrected by visual odometry, i.e. INS serves as a kind of battery analog for smoothing out visual odometry errors. So, I don't know about zed cameras, but the d455 uses the Bosch BMI 055 IMU and it is a bad and inaccurate sensor, and the synchronization of the camera and IMU leaves much to be desired. In datasets with stereo cameras, imu synchronization is used based on the average value of the camera shutter, i.e. they make an interruption at the start of the camera shutter charge and then find the average time value at the end of the charge, and this value is closer and more accurate than in the d455 (at least it was before) a time stamp is used only from the start of the camera shutter charge. But the exposure time is not a constant, it changes depending on the strength of the ambient light, therefore the delay between the real movement in the frame and the one received from the imu also changes, which leads to errors until openvins adjusts to the new delays (lighting conditions).

goldbattle commented 5 days ago

Could be a resolution, calibration, IMU quality, timesync, image quality, and other affects. Many parameters can affect. If the RS read out is small, then likely it should perform similar (up to some order) as a global shutter.