SteveMacenski / slam_toolbox

Slam Toolbox for lifelong mapping and localization in potentially massive maps with ROS
GNU Lesser General Public License v2.1
1.67k stars 524 forks source link

Karto strange memory usage behaviour #558

Closed zoidzilla closed 1 year ago

zoidzilla commented 1 year ago

Hello,

I am running Karto and I am monitoring the RES memory using the cpu_monitor package. I have used Karto individually (no global optimization and loop closures) and as a SLAM package (Karto SLAM, Karto with Ceres, Karto with g2o, Kator with GTSAM). What I am getting is the following figure (as extracted in Karto with Ceres as a back end, exhibiting similar behavior for all the other implementations). Furthermore, regarding SLAM Toolbox, I found in the following publication/thesis that exhibits similar Memory behavior since it is based on open Karto ( in figures 5.16, 5.17, 5.18 in Optimal Wheelchair Multi-LiDAR Placement for Indoor SLAM). Now, I understand that the memory usage builds up. What I don't understand is the memory behavior after 70MB (it continually allocates and frees a lot of memory blocks). So far, I have not obtained any good explanation why this happens. Can you help?

Best regards,

Karto_SLAM_Mem1

SteveMacenski commented 1 year ago

What if you turn off loop closures and anything related to the optimizer? I suspect it’s related to those

zoidzilla commented 1 year ago

Hello, I have already tried that as I am mentioning in my initial post: "I have used Karto individually (no global optimization and loop closures) and as a SLAM package", but it seams that it is completely independent from what back-end library you are using. My understanding so far it that this is totally due to Karto implementation. Now, since SLAM Toolbox tries to deal with the lifelong SLAM problem (and thus the total resources used), maybe you or any of the involved developers have encounter this behavior and have an explanation to it. In any case any help would be greatly appreciate it!

SteveMacenski commented 1 year ago

Change the occupancy grid generation frequency (or just dont generate one) - does that change the pattern proportionally? It could be that when we reallocate the occupancy grid to populate it after new data is processed is taking up a chunk of memory to store then its deleted when we publish it to the ether.

Both the regularity of the oscillation and the fact that it seems to grow quickly at first but then level off seems like the behavior I might expect from a SLAM trial in a space / map size.

I've definitely noticed this pattern (see charts in the readme) but I didn't concern myself much with it since it was still well below my requirements.

zoidzilla commented 1 year ago

Hi, that is actually a very probable explanation!! I could put a delay in the map update function to see what happens. Let me do some tests tomorrow and I will get back with my findings. Thanks a lot!

zoidzilla commented 1 year ago

Hello,

I did some tests and it was exactly as you said! I have attached three figures below. In figure 1 I have completely deactivated the occupancy grid deletion at every map update (so we have a constant build up). In the second figure, I have introduced a 5 sec delay exactly after I delete the occupancy grid in the map update function (here I have not changed any other parameters for loop closure detection and thus the algorithm stopped detecting global loop closures due to the 5sec delayed distance the robot traveled). Figure 3 I have introduced a 5 sec delay before I delete the occupancy grid map. The only strange issue here is that theoretically I would have expected to have very small negative peaks that would reflect the occupancy grid deletion. It seams that there is some kind of overlap when creating a new occupancy grid map immediately after when deleting one. In any case, I now understand why this is happening and thank you very much for your assistance! All the best!

Fig11_no_grid_delete Figure 1 Fig11_5s_delay_after_delete Figure 2 Fig11_5s_delay_before_delete Figure 3