Suggestions about running on my own datasets

[dane-wang]

Hello there, Great work! The result results in the paper and demo video are really impressive. I have tried to run DM-VIO on my own datasets, but it doesn't perform very well and I would appreciate some suggestions about how to make it work better. Thanks in advance.

Here are some problems I am facing:

1. For the camera.txt, the resolution of my dataset is 320x240. If I use the crop mode, it will cause some distortion (kinda serious distortion) even if I output the same resolution. I believe it is because a new camera matrix is calculated and used for undistortion. I don't understand why we do this, and if it is normal that I had some obvious distortion? By the way, if I use none mode, the odometry will drift badly and full mode will fail directly (it is commented as todo in the code).
2. IMU takes a long time to finish the initlizalition for some datasets. It needs only a few seconds for some datasets, but can't finish initlizalition even after the whole dataset is over (around 70 to 90s). Do you think it is a calibration issue or the quality of the image or something else? Btw, I did try with your given imu noise config file, but it doesn't help.
3. The imu normalized error is also quite large for me (around 10k). I saw that you think it is probably caused by the imu-camera calibration and time synchronization. Do you think if there are some other possible reasons? Because I saw many people are having this problem.
4. I have an error called "Sophus exception: Quaternion is (near) zero!" for some of my datasets. Do you have any idea why this happen?

Any help would be appreciated. Thanks a lot.

[lukasvst]

1. 320x240 is relatively low resolution, but it should still work fine I think. For the distortion, are you talking about distorted lines? This should not happen and would mean that the camera calibration is bad. What can be the case for cameras with a very large field-of-view is that lengths get heavily distorted in the rectified image. Then it makes sense to replace "crop" with a custom line containing "fx fy cx cy 0". You can have a look at configs/tumvi_calib/camera02.txt as an example. Increasing fx and fy reduces the field of view but will often times yield better results, 0.2 0.2 0.499 0.499 0 is a good start.
2. You can increase the following three settings which determine with which variance the IMU initialization procedes:

   init_coarseScaleUncertaintyThresh: 5
   init_pgba_scaleUncertaintyThresh: 5
   init_pgba_reinitScaleUncertaintyThresh: 1

   (The values above are the defaults I use for the Realsense live demo.) To determine good values you can just grep all lines with "CoarseIMUInit normalized error" and see how small the printed variance needs to get until the scale value starts to make sense. But when the scale does not make sense for a long time even when you move the camera sufficiently and visual tracking is good, there is probably a problem with the IMU data or calibration.

3. For well-tracked sequences this error should be a lot smaller. Yes, the main reasons I can think of for this problem are imu-camera-calibration, time synchronization, too small IMU noise values (always start with my tumvi defaults first) or some other issue with the IMU data. I would be surprised if the VIO performs well when initialized with an error this large.
4. This sometimes happens when the system fails, but I think it should usually be the result of failed tracking, not the reason for it.

More generally speaking, you should first get the system running well with useimu=0 to ensure that camera calibration is good. Then you should ensure that the CoarseIMUInit normalized error gets reasonably low, otherwise the rest of the system will probably not work well.

[dane-wang]

Hi there, thanks for the suggestions. The customs line in camera.txt works much better for our dataset. Right now, the no imu mode works well, but it is still not that great when we add imu into the equation. There are some time synchronization issues I guess. I do have a suggestion about the project. When I use no imu mode, some frames are still ignored because they don't have corresponding imu measurements. It doesn't make sense since we are not using imu anyway. So probably disabled the imu check function when we are using visual only odometry. It should help the performance. Thank you.

[lukasvst]

I'm glad the tips on the camera.txt helped. I would still suggest to try to get the IMU mode working, as it is a lot more robust. Especially for custom systems (with camera and IMU bought separately) this can be quite tricky, as you need to get a lot of things right (time synchronization, calibration, etc.), but it's worth it.

One advantage of the CoarseIMUInitializer is that it is a relatively simple concept (it just fixes the poses from the visual system and only optimizes IMU variables), meaning that it is most likely bug-free and can be used to debug most problems with camera-IMU.

Thanks for the suggestion, currently the visual-only-mode is only partially supported in the ROS driver. I might lift this in the future, but for now the visual-only-mode is mostly there for debugging, and it only pertains the first couple of images.