search

ros2 / rmw_fastrtps

Implementation of the ROS Middleware (rmw) Interface using eProsima's Fast RTPS.
Apache License 2.0
157 stars 117 forks source link

Large number of consecutive messages queued/delayed when using RMW Fast-RTPS 6.2.2 (with Fast DDS 2.6.3) #656

Closed nvcyc closed 1 year ago

nvcyc commented 1 year ago

Bug report

Required Info:

Steps to reproduce issue

With the RMW Fast-RTPS 6.2.2 (which is the version currently associated with Humble when installed using rosinstall_generator), when continuously publishing messages from a publisher to a subscriber, we observed that intermittently a large number of consecutive messages may be queued and not delivered to the subscriber immediately for unknown reasons at random times

When such a symptom happened, we also observed that some of those queued messages can be successfully delivered to the subscriber (i.e., the subscriber’s callback is called) after stopping publishing messages and starting to publish new messages.

This symptom seems to happen more often when the message publisher rate is high, yet it can still be observed occasionally when the rate is low.

This issue is not observed once we downgrade RMW Fast-RTPS to 6.2.1.

Expected behavior

With RMW Fast-RTPS 6.2.1, messages are delivered as expected.

Actual behavior

With RMW Fast-RTPS 6.2.2, a large number of consecutive messages may be queued and not delivered to their subscriber at random times.

clalancette commented 1 year ago

@MiguelCompany @EduPonz Can you take a look?

fujitatomoya commented 1 year ago

@nvcyc do we have the same problem with rolling?

hemalshahNV commented 1 year ago

We have not tried on Rolling. Is there a version of the fastrtps-rmw this may already be resolved in and would it be compatible with Humble?

fujitatomoya commented 1 year ago

@hemalshahNV

Just thought there could be difference, if it does not happen in rolling, that would be also information track the problem.

(no issue) Fast-RTPS 6.2.1: https://github.com/ros2/rmw_fastrtps/tree/6.2.1 (has an issue) Fast-RTPS 6.2.2: https://github.com/ros2/rmw_fastrtps/tree/6.2.2

just curious, why you guys use these specific branches?

Ubuntu 20.04

that means you use Foxy as Tier I, right? Why not using foxy branch for rmw_fastrtps?

https://github.com/ros2/ros2/blob/3bc9b4cd9a6661f09ae5f271d4f5f4a3b11cccd9/ros2.repos#L286-L289

Fast-RTPS 6.2.2 (which is the version currently associated with Humble

This is incorrect.

EduPonz commented 1 year ago

This is incorrect.

I guess the OP is referring to rmw_fastrtps 6.2.2; the corresponding Fast DDS version should be v2.6.2. Is that correct @nvcyc? In that case, it'd be great if you could update the ticket description

Mario-DL commented 1 year ago

Yes, I think that @nvcyc was referring to the v6.2.2 version of rmw_fastrtps as @EduPonz was pointing. Fast DDS should be 2.6.2.

Please, check if the issue persists using PR 3195 in Fast DDS

nvcyc commented 1 year ago

Thanks for helping me clarify the versions in discussion. Yes, I was initially referring to the RMW FastRTPS rmw_fastrtps_cpp version 6.2.2 in this repository. The related Fast DDS fastrtps version was 2.6.3 when we observed the issue.

I just tested Fast DDS 2.6.4 that contains the PR @Mario-DL mentioned with RMW FastRTPS 6.2.2 and I no longer observe the frame drop/delay issue.

I'll rebuild our environment with rosinstall_generator (that now pulls fastrtps version 2.6.4-1) and verify again before closing this ticket.

clalancette commented 1 year ago

I'm going to assume that this was fixed by 2.6.4, and close this out. If that is not the case, please feel free to reopen.