Realsense ZR300 with IMU / Fisheye support

[sjobeek]

I'm curious whether the developers had a look at the new Realsense ZR300 with integrated IMU and fisheye lens. I have a feeling that rtabmap integration would result in an extremely robust localization / mapping solution.

https://click.intel.com/intelr-realsensetm-development-kit-featuring-the-zr300.html

Any thoughts or plans on this front? I have a few of these sensors, and the tracking is quite solid using the intel-provided (binary) SLAM library. https://software.intel.com/sites/products/realsense/slam/developer_guide.html

[matlabbe]

This is indeed a great sensor! It looks like a standalone version of the Tango's tracking module. It could be easily integrated in rtabmap, as I can see in this video: https://www.youtube.com/watch?v=XPlTrIjvP0g. Gonna look to pre-order one.

thx for the info! Mathieu

[matlabbe]

Just read about project Euclid (small computer integrating ZR300): http://linuxgizmos.com/intel-euclid-a-brain-vision-sensors-and-hotspot-module-for-robots/, it looks great too, though there seems no release date yet.

[lancelocey]

The ZR300 is now available. Has anyone attempted to integrate the ZR300 into RTABMap?

I've tried, but I cannot seem to get a /depth_registered topic from the camera. Has anyone else run into this same problem? Here's the output of rostopic list: /camera/depth/camera_info /camera/depth/image_raw /camera/depth/image_raw/compressed /camera/depth/image_raw/compressed/parameter_descriptions /camera/depth/image_raw/compressed/parameter_updates /camera/depth/image_raw/compressedDepth /camera/depth/image_raw/compressedDepth/parameter_descriptions /camera/depth/image_raw/compressedDepth/parameter_updates /camera/depth/image_raw/theora /camera/depth/image_raw/theora/parameter_descriptions /camera/depth/image_raw/theora/parameter_updates /camera/depth/points /camera/depth/points/image /camera/depth/points/image/compressed /camera/depth/points/image/compressed/parameter_descriptions /camera/depth/points/image/compressed/parameter_updates /camera/depth/points/image/compressedDepth /camera/depth/points/image/compressedDepth/parameter_descriptions /camera/depth/points/image/compressedDepth/parameter_updates /camera/depth/points/image/theora /camera/depth/points/image/theora/parameter_descriptions /camera/depth/points/image/theora/parameter_updates /camera/depth/points/image_raw /camera/depth/points/image_raw/compressed /camera/depth/points/image_raw/compressed/parameter_descriptions /camera/depth/points/image_raw/compressed/parameter_updates /camera/depth/points/image_raw/compressedDepth /camera/depth/points/image_raw/compressedDepth/parameter_descriptions /camera/depth/points/image_raw/compressedDepth/parameter_updates /camera/depth/points/image_raw/theora /camera/depth/points/image_raw/theora/parameter_descriptions /camera/depth/points/image_raw/theora/parameter_updates /camera/driver/parameter_descriptions /camera/driver/parameter_updates /camera/fisheye/camera_info /camera/fisheye/image_raw /camera/fisheye/image_raw/compressed /camera/fisheye/image_raw/compressed/parameter_descriptions /camera/fisheye/image_raw/compressed/parameter_updates /camera/fisheye/image_raw/compressedDepth /camera/fisheye/image_raw/compressedDepth/parameter_descriptions /camera/fisheye/image_raw/compressedDepth/parameter_updates /camera/fisheye/image_raw/theora /camera/fisheye/image_raw/theora/parameter_descriptions /camera/fisheye/image_raw/theora/parameter_updates /camera/imu/data_raw /camera/ir/camera_info /camera/ir/image_raw /camera/ir/image_raw/compressed /camera/ir/image_raw/compressed/parameter_descriptions /camera/ir/image_raw/compressed/parameter_updates /camera/ir/image_raw/compressedDepth /camera/ir/image_raw/compressedDepth/parameter_descriptions /camera/ir/image_raw/compressedDepth/parameter_updates /camera/ir/image_raw/theora /camera/ir/image_raw/theora/parameter_descriptions /camera/ir/image_raw/theora/parameter_updates /camera/ir2/camera_info /camera/ir2/image_raw /camera/ir2/image_raw/compressed /camera/ir2/image_raw/compressed/parameter_descriptions /camera/ir2/image_raw/compressed/parameter_updates /camera/ir2/image_raw/compressedDepth /camera/ir2/image_raw/compressedDepth/parameter_descriptions /camera/ir2/image_raw/compressedDepth/parameter_updates /camera/ir2/image_raw/theora /camera/ir2/image_raw/theora/parameter_descriptions /camera/ir2/image_raw/theora/parameter_updates /camera/nodelet_manager/bond /camera/rgb/camera_info /camera/rgb/image_color /camera/rgb/image_color/compressed /camera/rgb/image_color/compressed/parameter_descriptions /camera/rgb/image_color/compressed/parameter_updates /camera/rgb/image_color/compressedDepth /camera/rgb/image_color/compressedDepth/parameter_descriptions /camera/rgb/image_color/compressedDepth/parameter_updates /camera/rgb/image_color/theora /camera/rgb/image_color/theora/parameter_descriptions /camera/rgb/image_color/theora/parameter_updates /camera/rgb/image_mono /camera/rgb/image_mono/compressed /camera/rgb/image_mono/compressed/parameter_descriptions /camera/rgb/image_mono/compressed/parameter_updates /camera/rgb/image_mono/compressedDepth /camera/rgb/image_mono/compressedDepth/parameter_descriptions /camera/rgb/image_mono/compressedDepth/parameter_updates /camera/rgb/image_mono/theora /camera/rgb/image_mono/theora/parameter_descriptions /camera/rgb/image_mono/theora/parameter_updates /camera/rgb/image_proc_debayer/parameter_descriptions /camera/rgb/image_proc_debayer/parameter_updates /camera/rgb/image_proc_rectify_color/parameter_descriptions /camera/rgb/image_proc_rectify_color/parameter_updates /camera/rgb/image_proc_rectify_mono/parameter_descriptions /camera/rgb/image_proc_rectify_mono/parameter_updates /camera/rgb/image_raw /camera/rgb/image_raw/compressed /camera/rgb/image_raw/compressed/parameter_descriptions /camera/rgb/image_raw/compressed/parameter_updates /camera/rgb/image_raw/compressedDepth /camera/rgb/image_raw/compressedDepth/parameter_descriptions /camera/rgb/image_raw/compressedDepth/parameter_updates /camera/rgb/image_raw/theora /camera/rgb/image_raw/theora/parameter_descriptions /camera/rgb/image_raw/theora/parameter_updates /camera/rgb/image_rect /camera/rgb/image_rect/compressed /camera/rgb/image_rect/compressed/parameter_descriptions /camera/rgb/image_rect/compressed/parameter_updates /camera/rgb/image_rect/compressedDepth /camera/rgb/image_rect/compressedDepth/parameter_descriptions /camera/rgb/image_rect/compressedDepth/parameter_updates /camera/rgb/image_rect/theora /camera/rgb/image_rect/theora/parameter_descriptions /camera/rgb/image_rect/theora/parameter_updates /camera/rgb/image_rect_color /camera/rgb/image_rect_color/compressed /camera/rgb/image_rect_color/compressed/parameter_descriptions /camera/rgb/image_rect_color/compressed/parameter_updates /camera/rgb/image_rect_color/compressedDepth /camera/rgb/image_rect_color/compressedDepth/parameter_descriptions /camera/rgb/image_rect_color/compressedDepth/parameter_updates /camera/rgb/image_rect_color/theora /camera/rgb/image_rect_color/theora/parameter_descriptions /camera/rgb/image_rect_color/theora/parameter_updates /rosout /rosout_agg /tf /tf_static

[matlabbe]

The ZR300 has been integrated in the standalone version of rtabmap, not ROS. To use it with ROS, it should be similar to R200 example on this page: http://wiki.ros.org/rtabmap_ros/Tutorials/HandHeldMapping

The visual inertial odometry (VIO) feature has been tested only with the standalone, for which realsense_sample_ros package should be installed before building rtabmap (so that rtabmap can detect realsense slam library needed for VIO). The VIO option will be enabled in Preferences->Source->RealSense if so.

EDIT: I updated the tutorial on ROS for ZR300 (without VIO): http://wiki.ros.org/rtabmap_ros/Tutorials/HandHeldMapping

cheers, Mathieu

[mpallewa]

I compiled the standalone version of rtabmap on Euclid and encountered a segfault every time the mapping was stopped,

[Thread 0x7fff94d0c700 (LWP 2785) exited]

Thread 11 "rtabmap" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff970d3700 (LWP 2668)] 0x00007fffd5ce85dd in motion::MotionHalListnerAdapter::MotionHAL_onNotify(motion::MotionHAL::NotifyMsg const&) () from /usr/lib/libslimAPI.so

The issue can be temporarily avoided by commenting out the deletion of the _camera object in the destructor but will obviously result in a memory leak. What's puzzling is that the issue is only seen on Euclid and not on a x86_64 Ubuntu laptop with ZR300. (Euclid has ZR300 hardware). I have posted these details to the Intel Euclid forum as well.

Thanks,

Methlal

Thread 1 "rtabmap" hit Breakpoint 1, rtabmap::CameraThread::~CameraThread ( this=0x40e76c0, __in_chrg=) at /home/euclid/rtabmap/corelib/src/CameraThread.cpp:73

73 join(true); (gdb) list 68 } 69
70 CameraThread::~CameraThread() 71 { 72 UDEBUG(""); 73 join(true); 74 // if(_camera) 75 // { 76 // delete _camera; 77 // }

[matlabbe]

The bug would be when releasing realsense resources or in their destructors here:

dev_->stop(rs::source::all_sources); // rs::device * dev_;

or

delete ctx_; // rs::context * ctx_;

or

slam_->flush_resources(); // rs::slam::slam * slam_;
delete slam_;

Maybe there is a new method (in realsense library) that should be called before deleting the realsense objects, in particular for Euclid.

cheers, Mathieu

[mtee]

Hey Mathieu, a quick question regarding zr300 IMU data. I have a zr300 recording in form of a rtabmap-database, where the IMU data went completely wrong at a certain stage, probably due to overheating of the sensor or something. It resulted in an accumulation of translational drift of few meters during few seconds, so I'd like to reprocess the part of the database without IMU data, just by using visual odometry. I tried setting "ignore odometry saved in database" to true, however rtabmap still used the IMU data as far as I can tell, i.e. the drift still occured. I tried searching for the IMU data in the database by using DB Browser for SQLite but could not find it. Do you have any hints for me? An obvious and dirty hack would be for me to go into source code where the IMU data is fused and check for certain frame id's range and ignore the IMU if the id is higher than where the IMU malfunctioned.

[matlabbe]

Normally with "ignore odometry saved in database", odometry should be recomputed with only RGB-D data, no IMU. However, the frame rate recorded by rtabmap may not be high enough for good visual odometry. Another approach can be to open database in DatabaseViewer, open View->Constraints view and scroll the neighbor links up to erroneous one, then try Refine to recompute the transform between the 2 frames.

IMU data is not saved in database, but included in the pose saved in Node table. If you can share the database, I may look more in depth if we can actually correct this error.