Map update & Control loop loop missed its desired rate.

[asimay]

with new code and old code, it occurs many loop warning like below.... please check the screenshot below, thanks!

And after check code I think I find out a temporary solution, maybe it is also the root cause: in costmap_2d_ros.cpp file: ROS_WARN("Map update loop missed its desired rate of %.4fHz... the loop actually took %.4f seconds", frequency, at line 409, r.sleep(); if (r.cycleTime() > ros::Duration(1 / frequency)) , the loop time must have some issues, maybe not correct.

I test the map loop time in function, most of them is actually very small, and can satisfy our needs.(based on small map 10*10m, not test large map.) if we comment the r.sleep(); the output is very good, and almost no loop warning output. so the cycleTime's calculation must be not correct. if no wrong, why we comment the sleep(), the function act good and loop warning disapper?

and with the same method, we can remove the controller loop warning in the same way. it acts good finally.

could you please check above bug? thanks!

[alexhenning]

If you look at how sleep and cycleTime are implemented, you'll see that the cycle time only gets updated when sleep is called. Not calling sleep may avoid the prints, but it is unlikely that it is actually helping your performance.

[mikeferguson]

It appears you are in simulation -- is the parameter "use_sim_time" actually set to true?

[asimay]

Dear @alexhenning ,yes, i look through the sleep prototype of ros. but it indeed help me on not print the loop warning info if i comment the sleep code. so from the code logic, we can verify that the cycle time is not exceed the frequency time, so the loop warning info not be printed out.

Dear @mikeferguson , the parameter "use_sim_time" is set to true, all the costmap's parameter is tuned well according to tutorials. and from #432 we also see the author also has same loop warning info printed out. and also please these old info: http://answers.ros.org/question/194980/someone-has-this-error-map-update-loop-missed-its-desired-rate-of-50hz/ http://answers.ros.org/question/195523/warning-map-update-loop-missed-its-desired-rate-keeps-on-comming/

please see below screenshot:

[corot]

mmm... this is very interesting... because starting from last week, the performance of my move_base configuration, with Stage simulation, has degraded hugely:

It fails to keep updating both local costmap and controller even at 5 Hz, in a rather powerful PC. Before, it failed to keep the requested Hz from time to time, but not systematically and by far as now.

I have absolutly no idea what happened, and what I changed or installed in my PC. I was tinkering with navigation stack sources, but now both sources and deb packages (indigo) behaves the same.

@asimay , do you experience the poor performance only in simulation?

[asimay]

dear @corot , not only the simulation case but also real time env case get the same result. I have already changed the code, commented ther.sleep() in costmap and control related source files, just two modifications, and it can avoid the infinite loopping print info. Maybe something strange blocked it during the function sleep time.

[corot]

I think I found where is the problem, but not at all the cause. The inflation layer takes almost all the update time for the maps, and in particular calls to pop on the priority_queue (this line) I have absolutely no idea why, but the fact is that whereas the full inflation layer update takes around 0.4 seconds in my scenario, calls to this pop amount for 0.3 seconds, 75% percent!!! Reading a bit about the std::priority_queue, pop has a complexity of O(log n), and it is called for roughly half of the cells. Could this justify the 0.3s ? No idea. :question::question::question:

[corot]

I'm so intrigued with this that I remade the inflation algorithm without using the priority_queue. In my scenario I get a x3 boost in costmap update performance, but as it's clear to me that there's something definitely wrong with my ROS-navigation-stage workspace, I would really appreciate if someone can give the change a try:

https://github.com/corot/navigation/tree/speedup_costmaps

Also, I use C++11. Hope it's not an issue in case I PR the change. @mikeferguson, @DLu, any comment on this?

[corot]

I have played around with the changes a bit more, and all seems to work fine, also scaling well to higher resolutions and inflation radius. After profiling my changes, the new biggest CPU eater is the map lookup for distances on enqueue method (you need to check my changes to understand what I mean). Maybe someone with better understanding of STL containers can give an extra speedup, knowing this.

[AlexReimann]

Indigo only with C++03

:/

[hcv1027]

Hi @corot, I just merge your pull request "Speedup (~60%) inflation layer update #525" But it still shows the warning log: "Map update loop missed its desired rate of 5.0000Hz... the loop actually took 0.7620 seconds". Does your merge request #525 can fix this problem or it's another problem?

[corot]

Hi @hcv1027, well, you should see the " loop actually took xxx seconds" figure decreased after merging, for sure, and this decrese should be easily noticeable. But not necessarily remove the WARN, if your update rate is too high for your CPU and costmap size/resolution.

Just to be sure, can you double check that you are running the compiled version? If you don't see any performance improvements, maybe you are still using the .deb packages!

[mikeferguson]

I just tested on one of our robots and I don't see huge delays from debs -- however, if I build source without -DCMAKE_BUILD_TYPE=Release, well, that's a different story, nothing runs in allotted time. I don't recall having to specifically call out Release in the past... but can't figure out how that would have changed.

[corot]

Yes, I noticed the same effect, so we always compile on Release. That said, compile on Debug is the right thing to do by default when running the code. But maybe a warning and the advice to compile on Release if performance is needed, either on github or wiki, would make sense.

[tiralonghipol]

Hi. I don't know if anyone still interested, but I've been deeply searching for a solution to this problem. All the cited solutions did not worked for me, so I got one on my own..

The only thing that would drastically produce a lower load in terms of CPU and computational power, in my case, was to act in the number of samples (forward simulation - vx and vy samples values).

Hope it helps!

[HappySamuel]

I'm so intrigued with this that I remade the inflation algorithm without using the priority_queue. In my scenario I get a x3 boost in costmap update performance, but as it's clear to me that there's something definitely wrong with my ROS-navigation-stage workspace, I would really appreciate if someone can give the change a try:

https://github.com/corot/navigation/tree/speedup_costmaps

Also, I use C++11. Hope it's not an issue in case I PR the change. @mikeferguson, @DLu, any comment on this?

Hi @corot Is your branch of speedup costmaps still available? Because it seems like i couldn't find it.

Hi @tiralonghipol What're the value you used on vx & vy samples (forward simulation)?

Best, Samuel

[wanghuohuo0716]

I meet the same situation, my cpu is intel-i7-8750H

And the move_base thread only cost 56% of one core!

Is there any actions to help me speed up the update rate of map and controll?

[wanghuohuo0716]

I'm so intrigued with this that I remade the inflation algorithm without using the priority_queue. In my scenario I get a x3 boost in costmap update performance, but as it's clear to me that there's something definitely wrong with my ROS-navigation-stage workspace, I would really appreciate if someone can give the change a try: https://github.com/corot/navigation/tree/speedup_costmaps Also, I use C++11. Hope it's not an issue in case I PR the change. @mikeferguson, @DLu, any comment on this?

Hi @corot Is your branch of speedup costmaps still available? Because it seems like i couldn't find it.

Hi @tiralonghipol What're the value you used on vx & vy samples (forward simulation)?

Best, Samuel

Dear Samuel

Do you fix it?@HappySamuel @corot

[corot]

I answered here

Also ensure you are compiling in Release