[Question] Working with high latency

[LKSeng]

We are using this camera driver for publishing the video as rostopics.

However, it was noticed that it takes about 1s for the video to appear when viewing the compressed rqt_image_view. Latency between the fiducial_images output from aruco_detect and the raw compressed image was not noticeable by human eye. Understandably, the robot steers out of control as the error compounds as it tries to steer dock with a mix of current and outdated tf (but as far as ROS is aware, they are current).

Are there sample configs for when the camera image latency is high?

There was also problems exiting the parallel correction loop, but I think that can be solved by setting a longer tf expiry time, and setting averaging sample(s) to 1.

[codebot]

It's pretty tough to tune a controller that would deal well with 1-second latency. It would need to either (1) drive the robot very slowly, or (2) predict the robot's future motions reasonably accurately.

I would suggest instead looking into why the latency is so high from the image, and seeing how that can be reduced. Are the images being processed on a different machine from where the images are being acquired? If so, what is the network like between them? Cheers

[LKSeng]

The camera in question is a consumer grade IP camera for general surveillance and we had issues getting the video (transferred via RTSP) to display under 1s latency even with direct Ethernet connection on VLC Player or FFplay.

We'll try the setup with another camera, as our working theory is that this latency likely camera dependent rather than caused by the network.

Thank you for sharing your thoughts on this issue.

[codebot]

oh, interesting. I haven't tried to use a camera like that with ROS in a long time. I guess if the camera is designed for security/surveillance, maybe it works harder at keeping the bandwidth low (lots of compression, etc) than in keeping latency low, or maybe it's just a low-power processor to keep costs low. As a "control" experiment, it might be worthwhile to use a known-low-latency camera like a high-end webcam directly connected over USB, since typically people care a lot about latency for video conferencing. Cheers