Stereo Support?

[ajsnyde]

Might be asking a lot, but is there any way to use this to calibrate stereo cameras with an IMU?

[urbste]

No at the moment there is not multi-camera support. I might add it in the future. What type of stereo camera do you have?

[ajsnyde]

Double GoPros of course! :smile: Trying to use SLAM+Lidar to map underwater caves.

[urbste]

Cool setup! I would like to help you with implementing stereo calibration. A couple of question:

• Are the cameras synchronized? Otherwise it is not straightforward to do stereo calibration. There would be a time delay that is also not fixed, but might change over time. Also for standard Stereo-SLAM the cameras must be synchronized (I saw in a screenshot that you are trying to use OrbSlam3
• Do you take still images or video in your application?
• Are they synchronized to the Lidar somehow?

[ajsnyde]

Cool setup! I would like to help you with implementing stereo calibration. A couple of question:

* Are the cameras synchronized? Otherwise it is not straightforward to do stereo calibration. There would be a time delay that is also not fixed, but might change over time. Also for standard Stereo-SLAM the cameras must be synchronized (I saw in a screenshot that you are trying to use OrbSlam3

Cameras are synchronized to within a frame of each other by flipping on a light and adding a R/L delay param to ORBSLAM. So I turn on both cameras, they start recording, I turn on the big lights, and in post I go frame-by-frame to find out the offset and apply it when ORBSLAM is run (I modified the library somewhat). I've yet to test how sync'd up the frames remain over time, but I don't imagine they'll drift too much. Definitely a see-as-I-go affair. If they do drift, I might automate something similar (flashing light every n seconds) and add some mechanism into the ORBSLAM library to handle it. Absolute worst case I'll go mono.

* Do you take still images or video in your application?

I take extremely high-resolution/FPS video (2.7k@120/240fps) on my dives with this setup, and downsize it in post. Processing time and storage is cheap. Diving costs time and money. If I ever want to crank up quality and params, I want to be able to reuse old footage. I think that the high FPS will also mitigate the effects of cameras not being 1000% sync'd up, but that's just a theory. My naive guess is that the accuracy of SLAM boils down to resolution/FPS. Higher resolution means better angular resolution for points. Higher FPS indirectly means less motion blur, more time resolution. Most applications seemingly aren't needing this scale of accuracy because they have GPS to fall back on for a ground-truth. In a cave, there is no GPS, so its extremely important to minimize drift over 10-60 minutes.

* Are they synchronized to the Lidar somehow?

Still working on acquiring the LiDAR. Short answer is that they will be sync'd to the LiDAR in the same mechanism. Light (activated by Raspberry Pi or Arduino) flashes, cameras capture that, LiDAR spins up, outputs from all devices get sync'd in post. The LiDAR data will be relative to the trajectory calculated by SLAM, forming a cloudpoint that will later be processed into a 3D map of the cave.

[urbste]

Just a thought about synchronisation. As you have two IMUs you could use the angular velcocities to synchronise the cameras from to time and in between motions just linearly interpolate the time offset (you will still have some jitter) . Basically how I initialize the calibration in this toolbox where I correlate visual and imu angular velcocities.

If you have fast motions, you might run in to inaccuracies and issues without hardware synchronisation. In addition GoPros have Rolling shutter sensors (however at 120/240 fps that might be negligible)

[ajsnyde]

Absolutely. Multi-IMU integration has crossed my mind, but that's 3 levels above where I am right now and am focusing on getting mono-inertial at this moment. I don't know where I'd start adapting ORBSLAM3 to use multiple IMUs. Having enough difficulty as it is just calibrating and running mono+IMU.

I've seen the rolling shutter effect when looking at individual frames at the moment that the sync light turns on sometimes, but it's kinda rare.