ros / ros_comm

ROS communications-related packages, including core client libraries (roscpp, rospy, roslisp) and graph introspection tools (rostopic, rosnode, rosservice, rosparam).
http://wiki.ros.org/ros_comm
761 stars 911 forks source link

roscpp PollManager busy loop from unexpected UDP data #2166

Open thomaskolBG opened 3 years ago

thomaskolBG commented 3 years ago

On melodic, and likely noetic as well based on a brief look, the roscpp PollManager::threadFunc function turns into a busy loop (resulting in 100% thread cpu utilization), when unexpected traffic is sent to the node's UDP port.

This happens because TransportUDP::socketUpdate only reads data from its UDP socket if it has read_cb_ set, which for a "normal" roscpp node does not happen upon node initialization even though a UDP port is always opened upon node initialization and its registered with the PollManager to continuously check for incoming data. Because TransportUDP::socketUpdate does not end up reading data from the socket, the poll_set_.update(100); call within PollManager::threadFunc's while loop will return immediately every time (=5us on the system i tested this on) because the file descriptor registered by TransportUDP will always have available data to read on it, hence turning the loop into a busy loop.

This can be reproduced very easily by starting rosout for example, or any other roscpp node that doesn't actually use UDP: 1) start roscore 2) find rosout's UDP port, e.g. by running sudo netstat -uplen | grep rosout 3) send a udp message to the roscpp process' udp port, e.g. by using nc 4) observe the process' cpu utilization being 100% now (e.g. htop) and its data on the udp socket never getting read (e.g. again sudo netstat -uplen | grep rosout where the first number after udp is the amount of data received that hasn't been read yet from the socket).

acolazo commented 1 year ago

I am having a similar problem but when subscribing to a topic using TransportHints().unreliable().

The udp packets are received by the corresponding process and put into the buffer. But ros never reads the buffer nor executes the callback function.

The thread ramps up to 100%. Rqt_graph shows the node is subscribed to the topic. The same thing happens with rosbag setting the UDP flag.

It works fine with TCP.

zerbobo commented 5 months ago

+1

Hugal31 commented 3 months ago

I have the same issue as @acolazo , subscribing using unreliable() works rarely, and when it doesn't it makes the process spin at 100%.

By looking at the tcpdump and the logs, it seems the XmlRpc requestTopic response is received but never processed by the client.

I think this deserves its own issue though.