Tuning 3D Cartographer to work with a HDL-32

[msy22]

I've got a HDL-32 mounted on a Jackal UGV (real, not simulated) and I'm trying to get Cartographer in 3D mode running on it. But unfortunately I can't get it to work. If anyone has a similar setup, or can provide any advice on how to get it working, I would be very appreciative.

I've documented the problem as best I can here on ROS answers

I've also opened a thread for discussion on the Cartographer google groups here in case that is anyone's preferred forum.

I suspect that my use case (one wide-angle 3D Lidar unit on a UGV) won't be that uncommon, so hopefully this will help others with similar setups.

[SirVer]

Try setting TRAJECTORY_BUILDER_3D.scans_per_accumulation = 1 instead of 160. Your sensor probably gives you one revolution per packet, while in our demo we have a few hundred laser returns per packet.

[rickeshtn]

Hi, @msy22 even I am working on the same thing. I tried your files from the ROS answers. I changed the robot param to the one from backpack_3d.urdf. But somehow I am not able to build the map. I have tried the solution from @SirVer as well. The map building doesn't happen in both 2D and 3D. Can you help me out on that? My settings in the cartographer.lua file was

provide_odom_frame = true, -- This should be false if using another package to generate odom (like ekf_localization) use_odometry = true, use_laser_scan = false, use_multi_echo_laser_scan = false,

Rest all is same from your lua files and I have included suggested change from Sirver. Still haven't got the map built.

@msy22 I guess the issue is with the robot_localization module. I was not able to get that running instead I used the one available inside cartographer. The submaps_list topic was always empty.

[msy22]

@rickeshtn When you say the map building doesn't happen in both 2D and 3D, do you mean that the cartographer_2d.launch file also doesn't work? Because the 2D version definitely should work if you're using it with the cartographer_configuration_2d.lua file. I've added a short video to my ROS question under "Video example of working 2D Cartographer". That is what you should see in RViz when you run the 2D launch file.

Regarding the map using 3D cartographer, could you post an image or two of what you see in RViz? It might help visually debug what's going on.

Unfortunately my computer broke shortly after posting that ROS question, and I'm still waiting for it to be fixed. So I haven't been able to make any more progress on the issue.

[msy22]

Got my computer back faster than expected!

@SirVer I've tried setting TRAJECTORY_BUILDER_3D.scans_per_accumulation = 1 but that change alone doesn't make it work. Some form of map is generated, but the trajectory is appalling so everything turns into a black smudge very quickly:

Regarding the number of scans per accumulation, how exact does this need to be? And since the input to Cartographer is a PointCloud2 message, not a raw UDP packet, I assume that points per unit input are determined by the drivers, rather than the raw UDP packets themselves?

If the Velodyne (or whatever LiDAR) drivers are important, the ones I'm using are these ones here. Looking at the source code it seems that each UDP packet contains 384 points, and that the drivers simply read a full packet them publish it.

Honestly, with so little documentation on what each individual setting does, it's proving to be virtually impossible to get the 3D version to work, let alone tune it.

[msy22]

@SirVer I think I may have misinterpreted the Rviz output when setting TRAJECTORY_BUILDER_3D.scans_per_accumulation = 1. That may well be the correct setting, but obviously the rest of the system is not well tuned for my particular setup. Having said that, I don't fully understand what is meant by "scans per accumulation" - does this basically mean "points per scan?"

At present, running through the same bag file in my ROS question with 3D cartographer produces results like this:

Is this likely to be a result of poor tuning on the local SLAM front end? Or poor tuning on the global back-end? I.e. should I be trying to tune the parameters in trajectory_builder_3d.lua or sparse_pose_graph.lua?

@rickeshtn Try turning off the global scan matching by setting SPARSE_POSE_GRAPH.optimize_every_n_scans = 0 in the .lua configuration file you're using. I've had marginally more success (see below) by turning it off and focusing on tuning the local SLAM.

I'd also recommend setting TRAJECTORY_BUILDER_3D.use_online_correlative_scan_matching = true and seeing if that improves the local SLAM for you.

[damonkohler]

I think your main issue is that your IMU has a angular velocity bias around the z-axis. It seems evident from the bag that the robot is not rotating. Yet, both your IMU and your odom (which I think is also being confused by IMU bias) show a constant rotational bias.

Cartographer has no support for filtering sensor biases. There is some support for correcting slight errors in alignment between LIDAR and IMU. Please reopen if still have issues after correcting your IMU bias.

[msy22]

@damonkohler Unfortunately that has less to do with my IMU itself, and has more to do with the steel exoskeleton of the building I'm in. I've always had problems with the IMU indoors. I'm trying to collect some outdoor data asap.

In the mean time, I've noticed that the global scan matching isn't very good at recursively going through the submaps to find better fits. If I can get it to re-match the submaps with a better orientation I think that will solve a significant portion of the problem. What are the specific parameters I should tune to improve this?

Thanks as always for your help.

[msy22]

@damonkohler Ignore everything I said about my IMU, it looks like you've spotted an important problem that I should have picked up on myself. There definitely was drift in that first bag file I posted:

And while I have re-calibrated the IMU twice since then, I've noticed that the angular velocity on the Z axis is biased, and the bias increases slightly over time - and unfortunately re-calibrating the IMU hasn't fixed that:

This will be an issue for me to look into separately. In the mean time, like I said, I have had more luck with Cartographer, but I'm still encountering issues where adjacent submaps obviously aren't being matched correctly:

Which happens indoors...

...and outdoors...

(notice the grand-stand thing at the top out the outdoor image). In your opinion, is this likely cause by my drifting IMU? and regardless of the IMU, do you think that this is something that can be corrected by correctly tuning the global submap matcher?

From what I've seen of Cartographer so far, I'd say that this should be fixable with proper tuning, and that's what my focus will be on in the coming days. But I thought I'd ask just in case.

And again, thanks for putting up with my barrage of information and incessant questions.

[rickeshtn]

@msy22 Thank you for the update. It will be really great if you could list the changes and updated files and share. I can take a look into it and get it working and so the one's who would be working on Velodyne 32.

[msy22]

@rickeshtn I've updated my ROS answers question with the latest .lua file I've been using and the important changes. I'll try to get a new bagfile up that doesn't have the IMU bias in it ASAP.

Regarding your setup, how are you getting on? I you still having issues with robot_localization and the submap topic appearing empty?

If you're having issues with robot_localization, that is a package I'm more familiar with so if you create an issue on ROS Answers and link me to it, I can help you out.

[damonkohler]

@msy22, the scan matching misses are quite likely due to errors in IMU data. You could try to counter the issue by increasing the number of loop closures attempted, increasing the search radius for loop closures, and increasing the importance of loop closures.

[msy22]

@damonkohler Yes, I'm becoming increasingly aware of how much of a problem my IMU is. I'm dealing with it in parallel to try and improve it's data quality.

Thanks for the tuning advice, do you have any suggestions on which parameters specifically I should be tuning?

I've mostly been figuring things out through trial and error. I've had a lot of improvement by tweaking SPARSE_POSE_GRAPH.optimize_every_n_scans to improve the trajectory smoothing, and increasing SPARSE_POSE_GRAPH.matcher_rotation_weight to stop new submaps from being added at an angle. But apart from that, I haven't had much success tinkering with the global SLAM/loop closure parameters.

[msy22]

I'd still appreciate some advice on global SLAM tuning parameters, since everything I've learned so far has been figured out my eye-balling the variable name, reading source code, and trial-and-error.

I have had a bit more success recently in resolving my IMU issues, which has helped a lot. I've updated the answers.ros question appropriately.

[SirVer]

@msy22 All we have for tuning is in #445. If you require more specific tuning help, could you reopen a new issue and follow the bug-template, so we have all information in one place?

[msy22]

The additional tuning notes introduced in # 445 were definitely very helpful. However I have reached a point where I'm not sure if further tuning would help, and if so what parameters to tune.

What bug template are you referring to?

[SirVer]

What bug template are you referring to?

The template text that is prefilled when opening a new issue: https://github.com/googlecartographer/cartographer_ros/issues/new

[msy22]

Ah, I presume "rosbag_validate" is a new feature? I am working on this and will post a new issue as soon as I can. Unfortunately I'm having issues re-installing Cartographer so this may take a while.

Specifically, Ceres is failing to build when I build it with ninja (see this issue here). I don't suppose ceres will still work with Cartographer if it's built without ninja? (i.e. by following the instructions given on the ceres page installation page)

[SirVer]

@msy22 Should work then too, as long as the version is recent.

[keir]

An update on Ceres: Sorry for the breakage; it has nothing to do with Ninja and just a minor issue on master. It's fixed now; please let me know if it's not working.

https://github.com/ceres-solver/ceres-solver/issues/317

[Duke-Allen]

@msy22 Hello, I am using a 16-lines LIDAR and IMU to build 3D map by cartographer. I tun the parameters and run it with no error, the results are as follows:

There is a big gap with the real environment. I think the map is messy. What is the problem of this result? How can I solve it? This is the validate result of my bag:

• E0227 22:30:57.092067 19056 rosbag_validate_main.cc:365] Average IMU linear acceleration is 10.6084 m/s^2, expected is [9.5, 10.5] m/s^2. Linear acceleration data should include gravity and be given in m/s^2. I0227 22:30:57.093359 19056 rosbag_validate_main.cc:398] Time delta histogram for consecutive messages on topic "/IMU" (frame_id: "imu"): Count: 37968 Min: 0.004133 Max: 0.016211 Mean: 0.010000 [0.004133, 0.005340) Count: 2 (0.005268%) Total: 2 (0.005268%) [0.005340, 0.006548) Count: 3 (0.007901%) Total: 5 (0.013169%) [0.006548, 0.007756) Count: 8 (0.021070%) Total: 13 (0.034239%) [0.007756, 0.008964) Count: 828 (2.180784%) Total: 841 (2.215023%) [0.008964, 0.010172) ################### Count: 35958 (94.706070%) Total: 36799 (96.921089%) [0.010172, 0.011380) Count: 345 (0.908660%) Total: 37144 (97.829750%) [0.011380, 0.012588) Count: 820 (2.159714%) Total: 37964 (99.989464%) [0.012588, 0.013796) Count: 1 (0.002634%) Total: 37965 (99.992096%) [0.013796, 0.015004) Count: 0 (0.000000%) Total: 37965 (99.992096%) [0.015004, 0.016211] Count: 3 (0.007901%) Total: 37968 (100.000000%) E0227 22:30:57.094650 19056 rosbag_validate_main.cc:382] Point data (frame_id: "rslidar") has a large gap, largest is 0.101409 s, recommended is [0.0005, 0.05] s with no jitter. I0227 22:30:57.094745 19056 rosbag_validate_main.cc:398] Time delta histogram for consecutive messages on topic "/rslidar_points" (frame_id: "rslidar"): Count: 3766 Min: 0.100213 Max: 0.101409 Mean: 0.100801 [0.100213, 0.100333) Count: 1 (0.026553%) Total: 1 (0.026553%) [0.100333, 0.100452) Count: 1 (0.026553%) Total: 2 (0.053107%) [0.100452, 0.100572) Count: 9 (0.238980%) Total: 11 (0.292087%) [0.100572, 0.100691) Count: 33 (0.876261%) Total: 44 (1.168348%) [0.100691, 0.100811) ############## Count: 2717 (72.145515%) Total: 2761 (73.313858%) [0.100811, 0.100931) ##### Count: 972 (25.809877%) Total: 3733 (99.123741%) [0.100931, 0.101050) Count: 25 (0.663834%) Total: 3758 (99.787575%) [0.101050, 0.101170) Count: 6 (0.159320%) Total: 3764 (99.946892%) [0.101170, 0.101289) Count: 1 (0.026553%) Total: 3765 (99.973450%) [0.101289, 0.101409] Count: 1 (0.026553%) Total: 3766 (100.000000%)

Here are my files: demo_rsldiar_3d.launch rslidar_3d.launch mydemo_3d.rviz rslidar_2d.urdf rslidar_scan_3d.lua my data bag Do you know the problem?

[msy22]

Hi @Duke-Allen. Unfortunately I stopped using Cartographer over a year ago, so I'm not familiar with it any more and it's highly likely that the code has changed since I last used it.

As you've probably discovered, Cartographer is quite opaque and hard to tune (part of the reason I stopped using it). So it's probably a tuning problem rather than a bug.

I suggest you re-post this in the Cartographer Github issue tracker as it's own issue, and add the "question" label and follow the instructions at the bottom of the Cartographer tuning page (link). Hopefully someone more up-to-date than me can help you out. Good luck!

[Duke-Allen]

@msy22 Thanks for your advice. I will try it.