search

magazino / move_base_flex

Move Base Flex: a backwards-compatible replacement for move_base
https://uos.github.io/mbf_docs/
BSD 3-Clause "New" or "Revised" License
424 stars 154 forks source link

Is anyone experiencing long term memory growth? #266

Closed boilerbots closed 3 years ago

boilerbots commented 3 years ago

If I keep MBF running for over 20 days the memory footprint is significantly larger than when it first started, eventually growing so large that the OOM (out of memory kernel module) will kill the process.

Is there anyone else that keeps their robot stack running for long periods of time in a production environment and experience the same thing or maybe they don't experience it?

spuetz commented 3 years ago

Thank you very much for your feedback. We will take a look at it.

corot commented 3 years ago

Hi @boilerbots, I have never noticed that. But bear in mind that MBF process also loads several plugins, and the plugins themselves can contain memory leaks (this actually happened to me with sbpl global planner). Some questions to narrow the options:

boilerbots commented 3 years ago

Hi @boilerbots, I have never noticed that. But bear in mind that MBF process also loads several plugins, and the plugins themselves can contain memory leaks (this actually happened to me with sbpl global planner). Some questions to narrow the options:

  • Are you loading any plugin other than those available on ROS navigation?

No not really, we have some custom changes to a few but nothing substantial.

  • Does the memory also grows without moving?

That is a good question, the robots are mostly busy but I am pretty certain that it does because some engineering units that sit idle still show the memory growth.

It is interesting that you mention SBPL because our suspicion was to look at SBPL, we enable GlobalPlanner and SBPL and a behavior tree makes decisions on which one to activate. I have not scraped the logs to try and correlate memory growth with frequency of the SBPL being activated but perhaps I should just remove it as a quick experiment.

boilerbots commented 3 years ago

We have resolved one long term memory leak but not related to MBF: https://github.com/magazino/tf_service/issues/16

reinzor commented 3 years ago

FYI: We are using MBF in production and we are not experiencing any memory issues. Process running for more than a month.

boilerbots commented 3 years ago

Perhaps the memory growth is unique to our system. Since we recently found a problem from using tf_service, I will close this and continue to investigate.

JohnTGZ commented 2 years ago

Hi, I have another observation from ROS Noetic. If you are using the default ROS Navigation stack installed via sudo apt-get install ros-noetic-navigation, you will face long term memory growth from using the 'global_planner/GlobalPlanner' plugin. This is especially apparent if you are using the '/get_path' action at a high frequency. I'm not sure what exactly was the bug, but I solved the long-term memory growth by building from scratch on the branch 'noetic-devel' of the ROS Navigation stack