RealSense R200: operation modes and baseline/depth calibration parameters...

[vazk]

Was anyone able to successfully use this sensor in orbslam2? I can see there are people who tried, did you manage to make it work? What are the color/depth operation modes that you use? And what are the Camera.bf, ThDepth and DepthMapFactor values that worked for you?

Thank you very much in advance!

[petern3]

I've managed to get it working with the SR300 and librealsense. It should be exactly the same for the R200 except for the parameters. If you can get the R200 viewing through an OpenCV window, you've done the hard part - you should be able to plug the cv::Mat objects straight into TrackRGBD.

You can access the intrinsics and baseline from the camera itself (search for 'extrinsics' and 'intrinsics' in rs.hpp (or rs.h) file). Otherwise you could use OpenCV to find the parameters.

fx, fy, cx, cy: come from the intrinsics. k1, k2, k3, p1, p2: can probably be left at 0? I'm using the ones left over from the TUM examples. width, height, fps: hopefully you defined these in enable_stream. If you use different resolutions/fps, you're liable to have issues. 640x480 @30fps works for me. bf: is the baseline (m) times the fx value. You can get the baseline from the extrinsics (between the colour and depth streams). Use the largest value (x should be way higher than the rest). rgb: again, you should have defined this during enable_stream. I would like to use rgb, but thanks to OpenCV, bgr is more convenient. ThDepth: apparently 50.0 should work. I've left mine at 40.0 from the example. DepthMapFactor: It seems to track okay regardless of this, though it changes how the visualization looks.

My values probably aren't the same as yours, but for the SR300:

Camera Parameters: 
- fx: 616.446
- fy: 616.446
- cx: 313.631
- cy: 240.362
- k1: 0.262383
- k2: -0.953104
- k3: 1.16331
- p1: -0.005358
- p2: 0.002628
- fps: 30
- color order: BGR (ignored if grayscale)

ORB Extractor Parameters: 
- Number of Features: 1000
- Scale Levels: 8
- Scale Factor: 1.2
- Initial Fast Threshold: 20
- Minimum Fast Threshold: 7

Depth Threshold (Close/Far Points): 0.800005

- baseline: 0.026 => bf = 12.329
- ThDepth: 40
- DepthMapFactor: 8000

I hope this is helpful!

[I0x0I]

Hi, I've run ORB_SLAM2 with F200 and R200, here is the code I used to get the parameters:

#include<iostream>
#include<stdio.h>
#include <librealsense/rs.hpp>

int main(int argc, char** argv)
{
    rs::context ctx;
    if(ctx.get_device_count() == 0) return EXIT_FAILURE;
    rs::device * dev = ctx.get_device(0);
    dev->enable_stream(rs::stream::depth, 640, 480, rs::format::z16, 30);
    dev->enable_stream(rs::stream::color, 640, 480, rs::format::rgb8, 30);
    dev->start();

    const float scale = dev->get_depth_scale();
    const float rgb_fx = dev->get_stream_intrinsics(rs::stream::color).fx;
    const float rgb_fy = dev->get_stream_intrinsics(rs::stream::color).fy;
    const float rgb_cx = dev->get_stream_intrinsics(rs::stream::color).ppx;
    const float rgb_cy = dev->get_stream_intrinsics(rs::stream::color).ppy;
    const float rgb_k1 = dev->get_stream_intrinsics(rs::stream::color).coeffs[0];
    const float rgb_k2 = dev->get_stream_intrinsics(rs::stream::color).coeffs[1];
    const float rgb_p1 = dev->get_stream_intrinsics(rs::stream::color).coeffs[2];
    const float rgb_p2 = dev->get_stream_intrinsics(rs::stream::color).coeffs[3];
    const float rgb_k3 = dev->get_stream_intrinsics(rs::stream::color).coeffs[4];
    const float baseline = dev->get_extrinsics(rs::stream::depth, rs::stream::color).translation[0];

    printf("Camera.fx: %.3f\nCamera.fy: %.3f\nCamera.cx: %.3f\nCamera.cy: %.3f\n\n",
            rgb_fx, rgb_fy, rgb_cx, rgb_cy);
    printf("Camera.k1: %.3f\nCamera.k2: %.3f\nCamera.p1: %.3f\nCamera.p2: %.3f\nCamera.k3: %.3f\n\n",
            rgb_k1, rgb_k2, rgb_p1, rgb_p2, rgb_k3);
    printf("Camera.bf:  %.3f\nDepthMapFactor: %.3f\n", fabs(baseline*rgb_fx), 1/scale);   
    return 0;
}

ThDepth: 50.0 works for both of the cameras but I still have no idea how to calculate it with the parameters above.

[vazk]

Thank you for your responses! One of the problems I was seeing was with the depth-map-factor, I wasn't sure about the units, and even in your code, I0x0I, the value it prints for DepthMapFactor is 1000, while the value for which the trajectory seems to be more or less reasonable is ~1000 times less. Can it be meter vs mm? What is the unit that dev->get_depth_scale() returns?

I have a different question regarding the type of the rsr200 streams to use, here are the types relevant to rgbd use-case:

• color
• depth
• rectified_color
• color_aligned_to_depth
• depth_aligned_to_color
• depth_aligned_to_rectified_color which types do you think should be used for color and depth streams? My guess is color and depth_aligned_to_color, but I may be wrong...

[petern3]

I did it initially with just the color and depth streams, and it worked OK. I have since changed to using color_aligned_to_depth because I assume it's more accurate. I imagine that using color and depth_aligned_to_color will work just as well, it depends on where you want the transform to be relative to.

As for the DepthMapFactor I'm using, I have no idea what it should be, but 1/get_depth_scale() gave me 1/0.000125 = 8000 which was in the same region as 5000 (in the example). Now that you mention the factor of 1000, it makes more sense because the Kinect measures in mm (as with many other RGBD sensors) but the realsense is in meters. The TUM dataset is probably in mm as well.

[I0x0I]

Take a look at the documentation of librealsense and the source code of ORB_SLAM2 might help you understand the DepthMapFactor ;-)

[vazk]

Actually, thanks for the link to librealsense document! I somehow haven't seen this document/explanation! The code, yeah, I know :)

petern3, my understanding of which stream should be aligned to which depends on which intrinsics is used in the system. For example I0x0I's example extracts intrinsics for 'camera' stream, so imho 'camera' and 'depth_aligned_to_color' streams should be used for rgbd slam.

[petern3]

I0x0I, thanks for the links (particularly jumping to the relevant sections). I had a poke around the ORB_SLAM code, but didn't look in the right places.

vazk, I thought the tracking was working okay before, but with your suggestion is even better!

[SubMishMar]

@petern3 Your answer is detailed. Thanks! Still, I don't get exactly how you calculate the bf? You have mentioned "baseline: 0.026 => bf = 12.329" but how is it possible? Given that the baseline is 0.026, the bf must be bf = baseline fx => bf = 0.026616.446 = 16.0276... isn't it? Please correct me where I am wrong.

Thanks!

[petern3]

@SubMishMar The value I gave for fx was for the colour stream, but I used the fx from the depth stream. I don't recall why sorry, so you could see which works better.

[SubMishMar]

@petern3 , thanks for your response. I will check which one work better.