Closed chanhhoang99 closed 1 day ago
@chanhhoang99
I try to reproduce this issue.
One question, I have no custom_action_interfaces.action.
Are the definitions for custom_action_interfaces.action
and example_interfaces.action
the same?
Hi @Barry-Xu-2018 , Could you check this https://docs.ros.org/en/foxy/Tutorials/Intermediate/Creating-an-Action.html and follow the steps. It's the same with this tutorial
@Barry-Xu-2018 i am not sure if this is the bug in rclpy yet, but i will go ahead to assign this to take a look.
Could you check this https://docs.ros.org/en/foxy/Tutorials/Intermediate/Creating-an-Action.html and follow the steps. It's the same with this tutorial
Thanks. Action message is almost the same.
My test code (use example_interfaces.action) is located at https://github.com/Barry-Xu-2018/bug_reproduce/commit/b52e6baef862e3871164faa7d6b56bfca261ca9f.
I ran it for over 5 hours (My environment is based on the latest code of rolling), and the memory usage changed as follows: After more than 5 hours
For client, resident memory is changed from 92M to 194M.
For server, Resident memory hasn't changed much (86M to 87M).
As reported, the memory usage on the client side is increasing significantly. So there's definitely an issue here with the client. I'll investigate further. @fujitatomoya
After investigating, I found the issue is related to the repeated calls to spin_until_future_complete.
When spin_until_future_complete returns, there may be other events that remain unprocessed.
The next time spin_until_future_complete is executed, there will be more unprocessed events again. This way, the pending events keep accumulating.
I find self._tasks keeps getting bigger.
https://github.com/ros2/rclpy/blob/a09a0312cf9e2bbe0f6dc4467c72060a2fb74ce5/rclpy/rclpy/executors.py#L184C9-L184C20
A not-so-good way to verify (It can't ensure that all events are processed.).
For client code, you can do
...
rclpy.spin_until_future_complete(action_client, future)
rclpy.spin_once(action_client, timeout_sec=0.01)
rclpy.spin_once(action_client, timeout_sec=0.01)
rclpy.spin_once(action_client, timeout_sec=0.01)
rclpy.spin_once(action_client, timeout_sec=0.01)
time.sleep(0.05)
...
In my explanation above, the word "event" should be more accurately referred to as "task".
Do you mean that the API is working as expect but the memory growth is due to user calling the API repeatedly ?
Do you mean that the API is working as expect but the memory growth is due to user calling the API repeatedly ?
Yes. This function only waits for the specified event to happen (it returns once it occurs). Other waiting events are not processed. The next time spin_until_future_complete is called, the unprocessed events get added to self._tasks, causing self._tasks to grow larger. Therefore, the memory usage we observe increases. But this isn't the memory leak. Following the demo example is a better way to handle it: https://github.com/ros2/demos/blob/ea3661152a87bc48e3f277ca8131c85a1a23d661/action_tutorials/action_tutorials_py/action_tutorials_py/fibonacci_action_client.py
Thanks alot. Let me try your suggestion.
@Barry-Xu-2018 thanks for checking this.
sorry i did not allocate the time to check the details but IMO this needs to be fixed in rclpy
, because i think that the following application code does nothing wrong? after all, my take here is API is not implemented as designed, unless user application needs to live with memory leak???
while rclpy.ok:
future = action_client.send_goal(xxx)
rclpy.spin_until_future_complete(action_client, future)
// do something with result...
@fujitatomoya
Agree with your thought.
After further checking, I found a bug which causes the number of unfinished tasks is increasing.
I will create a PR to fix it and explain the cause in that PR.
@chanhhoang99 I discovered a bug that increases memory usage. You can test this fix (#1374) with your code. In my environment, the client memory still increases a bit, but it's much less than before.
Bug report
Required Info:
Steps to reproduce issue
Using below Fibonacci action server and client. Action server:
Action client:
Expected behavior
Memory consumption from the client should not increase overtime
Actual behavior
Memory consumption from the client increases overtime
Additional information
Below is btop screen shot from the client.
Regards the function rclpy.spin_until_future_complete(action_client, future), below is the tracemalloc output:
I think the executor._tasks increases overtime. Is it expected to be growing in the example action client and server above ? FYI SteveMacenski