OpenVINS Multi-Camera Extension

[TheRealKaufmann]

Hi again,

I have been following the research papers regarding MIMC-VINS (Multi-IMU-Multi-Camera) and MC-VINS (Multi-Camera). It seems like that their implementations are based on OpenVINS. Unfortunately, I did not manage to implement the proposed methods into OpenVins, since there are relatively few infos about the actual implementation. Therefore my Question: What (key points) would have to be done in order to implement a simple Multi-Camera-Single-IMU solution like in the papers?

Or perhaps, would it be possible to share the code with us?

Thanks! Tom

[goldbattle]

At this point it is unlikely we will open source the MIMC-VINS code at this point, but this was built on top of the OpenVINS framework. This paper lays out all the key details for how one would go about implementing it. I am not sure of your use case, but the current code can work for synchronous multi-camera with a single IMU if that is all what you require.

[TheRealKaufmann]

Yes, that would be enough to start. Is there documentation for that already working code for synchronous multi-camera with a single IMU? Does it work with real cameras too or does it work only in simulation? Thanks!

[goldbattle]

Just continue to specify the topic / transforms / intrinsic as you do for stereo. You can also look at the main methods to see how we subscribe to topic to get a feel. It has not been fully tested, but if you give it a go feel free to post details (make sure mono/stereo works first before going to the third & forth camera).

<!-- bag topics -->
<param name="topic_imu"      type="string" value="/imu0" />
<param name="topic_camera0"  type="string" value="/cam0/image_raw" />
<param name="topic_camera1"  type="string" value="/cam1/image_raw" />
<param name="topic_camera2"  type="string" value="/cam0/image_raw" />
<rosparam param="stereo_pairs">[0,1]</rosparam>

<!-- camera intrinsics -->
<rosparam param="cam0_wh">[752, 480]</rosparam>
<rosparam param="cam1_wh">[752, 480]</rosparam>
<rosparam param="cam2_wh">[752, 480]</rosparam>
<param name="cam0_is_fisheye" type="bool" value="false" />
<param name="cam1_is_fisheye" type="bool" value="false" />
<param name="cam2_is_fisheye" type="bool" value="false" />
<rosparam param="cam0_k">[458.654,457.296,367.215,248.375]</rosparam>
<rosparam param="cam0_d">[-0.28340811,0.07395907,0.00019359,1.76187114e-05]</rosparam>
<rosparam param="cam1_k">[457.587,456.134,379.999,255.238]</rosparam>
<rosparam param="cam1_d">[-0.28368365,0.07451284,-0.00010473,-3.55590700e-05]</rosparam>
<rosparam param="cam2_k">[458.654,457.296,367.215,248.375]</rosparam>
<rosparam param="cam2_d">[-0.28340811,0.07395907,0.00019359,1.76187114e-05]</rosparam>

<!-- camera extrinsics -->
<rosparam param="T_C0toI">
    [
    0.0148655429818, -0.999880929698, 0.00414029679422, -0.0216401454975,
    0.999557249008, 0.0149672133247, 0.025715529948, -0.064676986768,
    -0.0257744366974, 0.00375618835797, 0.999660727178, 0.00981073058949,
    0.0, 0.0, 0.0, 1.0
    ]
</rosparam>
<rosparam param="T_C1toI">
    [
    0.0125552670891, -0.999755099723, 0.0182237714554, -0.0198435579556,
    0.999598781151, 0.0130119051815, 0.0251588363115, 0.0453689425024,
    -0.0253898008918, 0.0179005838253, 0.999517347078, 0.00786212447038,
    0.0, 0.0, 0.0, 1.0
    ]
</rosparam>
<rosparam param="T_C2toI">
    [
    0.0148655429818, -0.999880929698, 0.00414029679422, -0.0216401454975,
    0.999557249008, 0.0149672133247, 0.025715529948, -0.064676986768,
    -0.0257744366974, 0.00375618835797, 0.999660727178, 0.00981073058949,
    0.0, 0.0, 0.0, 1.0
    ]
</rosparam>

[TheRealKaufmann]

Oh wow, thank you so much! When I get some results, I will share them. Just one more question: With the parameter _stereopairs in your example, did you configure cam0 and cam1 as stereo and cam2 as mono?

[goldbattle]

Correct.

[TheRealKaufmann]

Thanks!

[robot-mh]

Oh wow, thank you so much! When I get some results, I will share them. Just one more question: With the parameter _stereopairs in your example, did you configure cam0 and cam1 as stereo and cam2 as mono?

Any progress？

[JimmyArwenHarry]

Just continue to specify the topic / transforms / intrinsic as you do for stereo. You can also look at the main methods to see how we subscribe to topic to get a feel. It has not been fully tested, but if you give it a go feel free to post details (make sure mono/stereo works first before going to the third & forth camera).

<!-- bag topics -->
<param name="topic_imu"      type="string" value="/imu0" />
<param name="topic_camera0"  type="string" value="/cam0/image_raw" />
<param name="topic_camera1"  type="string" value="/cam1/image_raw" />
<param name="topic_camera2"  type="string" value="/cam0/image_raw" />
<rosparam param="stereo_pairs">[0,1]</rosparam>

<!-- camera intrinsics -->
<rosparam param="cam0_wh">[752, 480]</rosparam>
<rosparam param="cam1_wh">[752, 480]</rosparam>
<rosparam param="cam2_wh">[752, 480]</rosparam>
<param name="cam0_is_fisheye" type="bool" value="false" />
<param name="cam1_is_fisheye" type="bool" value="false" />
<param name="cam2_is_fisheye" type="bool" value="false" />
<rosparam param="cam0_k">[458.654,457.296,367.215,248.375]</rosparam>
<rosparam param="cam0_d">[-0.28340811,0.07395907,0.00019359,1.76187114e-05]</rosparam>
<rosparam param="cam1_k">[457.587,456.134,379.999,255.238]</rosparam>
<rosparam param="cam1_d">[-0.28368365,0.07451284,-0.00010473,-3.55590700e-05]</rosparam>
<rosparam param="cam2_k">[458.654,457.296,367.215,248.375]</rosparam>
<rosparam param="cam2_d">[-0.28340811,0.07395907,0.00019359,1.76187114e-05]</rosparam>

<!-- camera extrinsics -->
<rosparam param="T_C0toI">
    [
    0.0148655429818, -0.999880929698, 0.00414029679422, -0.0216401454975,
    0.999557249008, 0.0149672133247, 0.025715529948, -0.064676986768,
    -0.0257744366974, 0.00375618835797, 0.999660727178, 0.00981073058949,
    0.0, 0.0, 0.0, 1.0
    ]
</rosparam>
<rosparam param="T_C1toI">
    [
    0.0125552670891, -0.999755099723, 0.0182237714554, -0.0198435579556,
    0.999598781151, 0.0130119051815, 0.0251588363115, 0.0453689425024,
    -0.0253898008918, 0.0179005838253, 0.999517347078, 0.00786212447038,
    0.0, 0.0, 0.0, 1.0
    ]
</rosparam>
<rosparam param="T_C2toI">
    [
    0.0148655429818, -0.999880929698, 0.00414029679422, -0.0216401454975,
    0.999557249008, 0.0149672133247, 0.025715529948, -0.064676986768,
    -0.0257744366974, 0.00375618835797, 0.999660727178, 0.00981073058949,
    0.0, 0.0, 0.0, 1.0
    ]
</rosparam>

Hello,

This kind of parameter setting seems to fit version "release v1.0" so I checkout to that branch. I set my launch file according to this. Two problems appear:

1. How should I set the parameter "max_cameras"? If I set it to 3 or 4, errors occured just like "[ERROR] [1667458794.988656197]: INVALID MAX CAMERAS SELECTED!!!".

2. If I set "max_cameras" to 2, the node only subscirbes cam0 and cam1.

Attached is my launch file. It would be more than better if you can provide a compact template launch file for multi-camera-single-imu example. Thanks so much.

Sincerely, Harry