D455 vs. D435i vs L515 for 3D Room scanning

[peta-peta]

Dear all, I want to scan rooms and create water tight meshes. I have started exploring intel realsense L515 with RTABMAP-Ros but I am not happy with the results. Mapping is very unstable and sensitive to daylight. Since I can see many successful applications using realsense D435i I am thinking to buy one, too, and use it directly with RTABMap. Today i learned that Intel has already released a new version D455 and I am wondering if any of you has some experience with it using RTABmap for mapping and odometry. Which one shall I buy, or do you suggest even a different camera or a combination of different devices for room scanning with RTABMap - Room has to be a watertight model with floor, walls, windows and ceiling.

Cheers Peter

[rlabs-oss]

Hi, I own the D455, L515 and D415 (but not the D435) - the L515 I got working with rtabmap following the earlier posts on the camera. For the D455 it works out of the box. - Once the IMU gravity is adjusted the D455 works very well. Ive used it to scan my house, outside structures, roads etc. The only issue I have with the cameras is the lack of portability.*1 I tried building rtabmap for the pi4 except it segfaults on start. Realsense viewer is working.

*1 Ive also built my own 3D lens for a canon EOS M as a portable rig, knock off of the Lumix 3D which I also own, neither will calibrate in rtabmap even with ffmpeg tricks to crop/resize/lenscorrect the stream before passing it to the v4l2 loopback (and opening that in rtabmap) -Looking into opencv2 and how that mangles reality next.

D455 top, L515 bottom - both scanned with rtabmap (and changes for the L515 code) https://urlme.net/dl/Sofa.jpg Not complete due to the laptop dropping frames using the L515. Mapping with the D455 https://urlme.net/dl/mapping.gif

[peta-peta]

Hi @rlabs-oss thank you very much for your detailed explanation. I will give the D455 a try. Regarding the L515, can you share your launch script with us? I suppose you run roslaunch rtabmap_ros with a launch script, right? Best wishes, Peter

[rlabs-oss]

Hi, I added the changes to the standalone app as suggested in this issue: https://github.com/introlab/rtabmap/issues/574 then rebuilt it. Main changes being the L500 Sensor: } else if ("L500 Depth Sensor" == module_name) { sensors[1] = elem; } else if ("Wide FOV Camera" == module_name)

and the depth sensor selection

if ((!ir_ && !irDepth_ && video_profile.stream_type() == RS2_STREAM_DEPTH) // L515 fix, use first available depth stream || (video_profile.width() == cameraWidth_ && video_profile.height() == cameraHeight_ && video_profile.fps() == cameraFps_)) //D400 series:

Currently I'm trying to figure out how to select the L515's infrared stream rather than the RGB stream which will be more useful in the dark. -M

[matlabbe]

For rtabmap standalone, a new commit has been added for L515: https://github.com/introlab/rtabmap/commit/3047b7da6b734628bdfe67eef1550d61a928a0d7

[peta-peta]

I tried the new commit for realsense L515. I executed ./rtabmap-rgbd_camera 11 and got this error [ERROR] (2020-10-20 20:04:53.761) CameraRealSense2.cpp:1374::captureImage() Missing frames (received 1)

[matlabbe]

Can you comment this line and see if it changes something?

[peta-peta]

I comment //sensors[i].set_option(rs2_option::RS2_OPTION_GLOBAL_TIME_ENABLED, 1); and recompiled cmake .. and make -j4 (or do I need to clean bin and build folders?)

Same Error.

[rlabs-oss]

I can detail the workarounds I made - 1st fix was to the gyro which if using Ubuntu20.04 the kernel stream sync isn't active.

This line to: `if(((iterA == iterB) && (stamp == iterA->first)) || isL500_)'

Since steams maybe de synced, I ignored them / pass them for the L500

And this line to ` if((!isL500_) && sensors[i].supports(rs2_option::RS2_OPTION_GLOBAL_TIME_ENABLED))'

Desired frames is also problematic if the IMU info doesnt arrive on time, comment it out ` UTimer timer; int desiredFramesetSize = 2; //if(isL500_) // desiredFramesetSize = 2;

More hacks really since if the kernel driver /stream sync is present the changes should not be needed.

[peta-peta]

@matlabbe and @rlabs-oss thank you very much! It works like charm. I ve tried already some tools under ros, but the results with rtabmap standalone are already very impressive and much more stable! I will keep you posted about my progress. Cheers Peter

[rlabs-oss]

The D455 + T265 quick room scan. The scanning was faster than shown in the video since I was post processing it and recording for upload. https://youtu.be/IDWoLNk0l5o

[samhoff20]

Hi guys, even after the new commit I don't see the l515 in my source list in the standalone app? How do I initialize the camera? I do have librealsense2 and the standalone app.

[rlabs-oss]

Manually setup the camera via the Select sensor driver/More Options button, or Detection - Select source - more options.

• Source Type RGB-D Grabber for RGB-D devices...
  • Driver Realsense2
  • IR Emitter enabled
  • IR mode unchecked
  • Use Depth... unchecked. 640 Stream width 480 Stream height 30 Hz rate In the standalone app should work.

[samhoff20]

Thank you! I seem to be having some realsense issues also but I will try this once I get them worked out!

[peta-peta]

I confirm that solution provided by rslabs-oss is working without problems. I was also able to use the standard settings of the D435 provided by RTABmap standalone tool for the L515. Settings by rlabs-oss work better though. Cheers Peter

[samhoff20]

So trying the custom source and the 415 both get me this output. Do you have any insight?

[samhoff20]

[rlabs-oss]

Hi, I have not had a chance to try 0.20.6 with the workarounds added but will check it now. -M

[rlabs-oss]

Leaving the gyro code alone (which will warn: "CameraRealSense2.cpp:409::getPoseAndIMU() Could not find gyro data to interpolate at image time 76198.835000 (earliest is 1604436689619.677734). Are sensors synchronized?" when using the L515 and Commenting out those lines to read //if(isL500_) // desiredFramesetSize = 3;

And adding the previous hack for global sync in this line So it reads: if((!isL500_) && sensors[i].supports(rs2_option::RS2_OPTION_GLOBAL_TIME_ENABLED))

Will enable the L515 at 640x480x30 in release 0.20.6 without causing the D415 to stop working. I tested this in dual mode T265 + D415. Single: D415, D455 (640x480x30, 60hz, and 1280x720x30) L515: 640x480x30

(librealsense 2.39.0 October 1st release, and all camera firmwares updated) Note: The above changes are "hacks" at best to work around the frame sync/global time issue on a standard kernel.

[samhoff20]

Thanks for the help. I tried the above edits but still can't initialize the camera. This is the output of the terminal running rtab (plus a few above that were cut off but can be seen in previous picture):

[rlabs-oss]

@samhoff20 Is the camera working with the realsense-viewer without error? Turn on all streams/modules in the viewer and watch the console log.

The difference for the changes is shown below.

diff ~/external-source/GitHub_RTAB/rtabmap/corelib/src/camera/CameraRealSense2.cpp corelib/src/camera/CameraRealSense2.cpp 1009c1009 < if(sensors[i].supports(rs2_option::RS2_OPTION_GLOBAL_TIME_ENABLED))

                   if((!isL500_) && sensors[i].supports(rs2_option::RS2_OPTION_GLOBAL_TIME_ENABLED))

1145,1146c1145,1146 < if(isL500_) < desiredFramesetSize = 3;

          //if(isL500_)
          //      desiredFramesetSize = 3;

check the output from ldd against rtabmap in the bin folder to show which version of librealsense2 its using.

ldd rtabmap | grep real librealsense2.so.2.39 => /usr/local/lib/librealsense2.so.2.39

[samhoff20]

This is the error I'm getting in the realsense terminal when trying to press play on rtabmap. realsense-viewer and having both depth sensor and rgb camera on (not motion module though). $ realsense-viewer 04/11 07:44:02,123 ERROR [139668014343936] (types.h:313) set_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device 04/11 07:44:02,124 ERROR [139668014343936] (global_timestamp_reader.cpp:239) Error during time_diff_keeper polling: set_xu(...). xioctl(UVCIOC_CTRL_QUERY) failed Last Error: No such device 04/11 07:44:02,247 ERROR [139667856922368] (types.h:313) xioctl(VIDIOC_DQBUF) failed for fd: 23 Last Error: No such device 04/11 07:44:02,247 ERROR [139667856922368] (backend-v4l2.cpp:1354) xioctl(VIDIOC_DQBUF) failed for fd: 23 Last Error: No such device 04/11 07:44:02,247 ERROR [139667659785984] (types.h:313) xioctl(VIDIOC_DQBUF) failed for fd: 27 Last Error: No such device 04/11 07:44:02,247 ERROR [139667659785984] (backend-v4l2.cpp:1354) xioctl(VIDIOC_DQBUF) failed for fd: 27 Last Error: No such device 04/11 07:44:02,247 ERROR [139667571734272] (types.h:313) xioctl(VIDIOC_DQBUF) failed for fd: 41 Last Error: No such device 04/11 07:44:02,247 ERROR [139667571734272] (backend-v4l2.cpp:1354) xioctl(VIDIOC_DQBUF) failed for fd: 41 Last Error: No such device 04/11 07:44:03,124 ERROR [139668014343936] (types.h:313) acquire: Cannot open '/dev/video0 Last Error: No such file or directory 04/11 07:44:03,125 ERROR [139668014343936] (global_timestamp_reader.cpp:239) Error during time_diff_keeper polling: acquire: Cannot open '/dev/video0 Last Error: No such file or directory

Then this last ERROR[139668014343936] repeats endlessly.

[samhoff20]

If I am just in the realsense viewer and I turn on both depth sensor and rgb turn on fine, but I get the following error when turning on the motion module if that is an issue: $ realsense-viewer 04/11 07:46:18,485 WARNING [140248687920960] (ds5-motion.cpp:480) IMU Calibration is not available, default intrinsic and extrinsic will be used. 04/11 07:46:18,492 WARNING [140247727863552] (backend-hid.cpp:681) HID set_power 1 failed for /sys/devices/pci0000:00/0000:00:14.0/usb2/2-7/2-7:1.7/0003:8086:0B64.0026/HID-SENSOR-200073.2.auto/iio:device0/buffer/enable 04/11 07:46:18,540 WARNING [140247711078144] (sensor.cpp:970) HID timestamp not found, switching to Host timestamps.

[rlabs-oss]

To clarify: realsense-viewer should not be open when using rtabmap they are both independent applications. Also are you using the D415 or the L515 ? The changes above are only related to the L515. The D415 should be unaffected or as functional as in the previous releases.

[rlabs-oss]

Then this last ERROR[139668014343936] repeats endlessly.

I do not experience that. With just realsense-viewer open (and not rtabmap) do you see the same error? If so it could be related to the USB bandwidth, USB port type, how the usb cable is plugged in -sounds strange but its better to plug into the usb port first then the camera. In realsense-viewer, with the camera connected click the Info button. is the "Usb Type Descriptor" 3.2? it should look something like this for the L515:

If any one of the L500 depth sensor, RGB camera and Motion module in realsense-viewer cannot be activated or generate lots of errors those should be resolved before using rtabmap.

[samhoff20]

I am using the L515.

Here is a snapshot of the info depth, rgb and motion turned on.

[Anderson191919]

Hello guys, I have an issue with the L515 on windows with rtabmap (v 0.20.6 from https://ci.appveyor.com/project/matlabbe/rtabmap/branch/master/artifacts). unfortunately I get following error(s) when starting on rtabmap (like samhoff20)

[Anderson191919]

[Anderson191919]

I have used these settings (#614 (comment)).

Thank you very much in advance. best regards

[Anderson191919]

[rlabs-oss]

Hi, Problem is related to lack of hardware timestamps from the realsense SDK for the frames (inc Gyro stream) which need a custom kernel module see:- https://github.com/IntelRealSense/librealsense/blob/master/doc/frame_metadata.md Its possible to check this in the realsense viewer. Top left, clilck 2D and turn all modules ON - The Motion Module Gyro stream window will indicate a red triangle error.

In RTAB-Map, if the rs2_option::RS2_OPTION_GLOBAL_TIME_ENABLED is used together with an expected number of frames (RGB, Depth, Gyro) then for the L515 and a non-custom kernel it will stall.

As a workaround, ==HACK, the checks for synced frames for the L515 can be disabled:
However this will cause gyro interpolation issues since the frame arriving from the gyro has a vastly different timestamp than the images thus this warning: "CameraRealSense2.cpp:409::getPoseAndIMU() Could not find gyro data to interpolate at image time 186557.513000 (earliest is 1606059194997.468750). Are sensors synchronized?" is seen constantly (set logging to fatal to ignore those.)

In summary the changes to the current version (as of 22nd November) are: rtabmap/corelib/src/camera/CameraRealSense2.cpp Line 1016 to 1016 if((!isL500_) && sensors[i].supports(rs2_option::RS2_OPTION_GLOBAL_TIMEENABLED)) and Lines 1152-1153 commented out 1152 //if(isL500) 1153 // desiredFramesetSize = 3;

Windows - Ive not tested but suspect it maybe the same issue based on previous posts. Hope this helps.

[Anderson191919]

[Anderson191919]

Thank you for your effort. Do you mean this error in the realsense viewer?

[Anderson191919]

I am not so good to make changes to cpp-files and then rebuild the project. Have you an idea to make the L515 run in rtabmap 0.20.6?

[Anderson191919]

I have read that some realsense modules are not calibrated from factory. Am I supposed to do it? What is your opinion?

[rlabs-oss]

Thank you for your effort. Do you mean this error in the realsense viewer? Not seen that, I get "CameraRealSense2.cpp:409::getPoseAndIMU() Could not find gyro data to interpolate at image time 324922.068000 (earliest is 1606066114724.759033). Are sensors synchronized?"

[rlabs-oss]

I am not so good to make changes to cpp-files and then rebuild the project. Have you an idea to make the L515 run in rtabmap 0.20.6?

Only by building a version with the changes as proposed

[matlabbe]

In the commit above, I added the changes from @rlabs-oss. There is a new option called "Global time sync" in Preferences->Source->RealSense2. By default it is checked, but if you uncheck it, it will do exactly @rlabs-oss proposed.

The windows build has started, just wait https://ci.appveyor.com/project/matlabbe/rtabmap/branch/master to finish the new artifact.

[matlabbe]

Please let me know if it works with a L515, as I cannot test it myself.

[Anderson191919]

I will definitely. Thank you very much :)

[rlabs-oss]

Please let me know if it works with a L515, as I cannot test it myself.

Tested & Confirmed to work (Linux Ubuntu 20.04)

Global Time sync On - CameraThread.cpp:154::mainLoop() no more images...

• as expected due to lack of sync.

Off - working with single warning "CameraRealSense2.cpp:442::getPoseAndIMU() As globalTimeSync option is off, the latest received gyro and accelerometer will be re-stamped with image time. This message is only shown once."

[matlabbe]

Thx @rlabs-oss, I could not test the warning message I added for imu/acc, it is exactly what I wanted. Note that I forgot that change in the other commit, causing imu not being published even if the warning was saying so.

[Anderson191919]

@matlabbe thank you, but still the same error :(

[Anderson191919]

[Anderson191919]

[Anderson191919]

Could it be really due to calibration issues? I have not calibrated it since I got it from factory. did you manually calibrate the l515? Thanks for your efforts

[matlabbe]

I need the first error in the log to debug. The one starting with "Given stream configuration is not supported by the device!"

No need to explicitly set 640x480x30 anymore in the Preferences dialog, in the code I just search for 640x480x30 when L500 is detected.

[Anderson191919]

[Anderson191919]

Here is my error log. Maybe I have to select manually the device-id? Anyway thank you for your efforts :)

[Anderson191919]

Or do I need to install Point Cloud Library, GTSAM or g2o?

[Anderson191919]

LogRtabmap.txt