Very large delay in image topics between machines in wlan

[RamsesReyes]

I'm working with the car version 2, with Ubuntu 16.04 running ROS kinetic. The ROS_MASTER is in a laptop, also with Ubuntu 16.04 running ROS kinetic, in a wlan in which the car is too. The problem is that there are tons of delay in the topic that publishes the camera images:

:~$ > rostopic delay /app/camera/rgb/image_raw/compressed
subscribed to [/app/camera/rgb/image_raw/compressed]
average delay: 1359.301
         min: 1358.945s max: 1359.655s std dev: 0.23105s window: 9

And the delay grows and grows even more. This is the ping from the laptop to the car:

:~$ > ping 192.168.1.103
PING 192.168.1.103 (192.168.1.103) 56(84) bytes of data.
64 bytes from 192.168.1.103: icmp_seq=1 ttl=64 time=2.44 ms
64 bytes from 192.168.1.103: icmp_seq=2 ttl=64 time=2.45 ms
64 bytes from 192.168.1.103: icmp_seq=3 ttl=64 time=1.98 ms

This is the bandwidth consumption of the topic:

:~& > rostopic bw /app/camera/rgb/image_color/compressed
subscribed to [/app/camera/rgb/image_color/compressed]
average: 365.30KB/s
       mean: 35.36KB min: 35.29KB max: 35.43KB window: 8

The clock time of the machines is sync with ntp. The delay is similar for the non-compressed image topics. The camera was initially taking images at 30Hz, then I set it at 10Hz but the problem persists. It is worth to mention that the delay is almost zero at starting the realsense nodelet, but it grows along the time.

I would really appreciate if you can tell me what I'm missing. Thanks in advance, kind regards.

[AutoModelCar]

Check how you publish the image, you should not cash the message. The second argument is the size of the publishing queue, make it 1.

[RamsesReyes]

Hello, thank you for your answer. The publisher is not mine, it is the one in the realsense_camera package in catkin_ws. I don't know how to check how is it being published. Concerning the subscriber, it has a queue size = 1.

[SGSSGene]

These lines https://github.com/IntelRealSense/realsense-ros/blob/ebc06d4be1800da82595a7ccb163f3b5b6e69b23/realsense_camera/src/base_nodelet.cpp#L374-L385 suggest that the publisher is also using a queue of size 1. (Not 100% sure if this is the correct line).

[RamsesReyes]

These lines https://github.com/IntelRealSense/realsense-ros/blob/ebc06d4be1800da82595a7ccb163f3b5b6e69b23/realsense_camera/src/base_nodelet.cpp#L374-L385 suggest that the publisher is also using a queue of size 1. (Not 100% sure if this is the correct line).

Hello @SGSSGene , thank you for your answer. Then, can we conclude that the queue size is not the problem here? Is there any else configuration that I could be missing?

[SGSSGene]

These lines https://github.com/IntelRealSense/realsense-ros/blob/ebc06d4be1800da82595a7ccb163f3b5b6e69b23/realsense_camera/src/base_nodelet.cpp#L374-L385 suggest that the publisher is also using a queue of size 1. (Not 100% sure if this is the correct line).

Hello @SGSSGene , thank you for your answer. Then, can we conclude that the queue size is not the problem here? Is there any else configuration that I could be missing?

Yes, I don't think the queue size is not the problem.

average delay: 1359.301

Does this mean 1.359s delay? or 1359s delay? Have you looked at the images and can also notice the delay visually? (This can tell us if it is an actual delay, or some timestamps are off)

[RamsesReyes]

I think it is 1359 seconds. Visually, the delay in the image is very low, it is almost imperceptible, so I think you're right with your timestamp theory. But, how can I confirm this? I tried the following: In a terminal I ran the following command to see the image timestamp:

rostopic echo /app/camera/rgb/image_rect_color/compressed/header

This is the last message that I got:

---
seq: 2018
stamp: 
  secs: 1629739377
  nsecs: 424555218
frame_id: "camera_rgb_optical_frame"
---

Then, in another terminal I ran at the same time: rostopic pub -s -r 1 /some_topic std_msgs/Time 'now'

This aims to publish the actual time in the ROS system, at the same time, I run the following to see it: rostopic echo /some_topic

This is the last message that I got:

---
data: 
  secs: 1629742364
  nsecs:  96251964
---

The messages that I'm showing in the image_rect_color header stamp and in 'now' stamp correspond approximately to the same instant, nevertheless, you can see that there is a discrepancy of about 2987 seconds, which indeed correspond to the delay shown by rostopic delay command. Then, that explains why those so large delays, but why the timestamps in the image messages are so out of date?

[RamsesReyes]

In the following comment of another forum: comment, they suggest me to reinstall the librealsense library with special instructions for Odroid XU4, but I'm afraid that the image in the eMMC could stop working at all. The eMMC card in the car has the image in the repository indicated in the AutoModelCar wiki, so I suppose that the librealsense in such an image is the one for the odroid XU4, isn't it?

[RamsesReyes]

Hello everybody. This problem has been solved reinstalling the realsense_camera package. I removed the package in the catkin_ws folder of the FUB repositories and then installed it by:

sudo apt install ros-kinetic-realsense-camera

That worked for me. Thank you very much @SGSSGene and @AutoModelCar for your support.

[SGSSGene]

@RamsesReyes This is awesome! I would have no idea where to continue searching for the cause of the issue :-)