search

osrf / subt

This repostory contains software for the virtual track of the DARPA SubT Challenge. Within this repository you will find Gazebo simulation assets, ROS interfaces, support scripts and plugins, and documentation needed to compete in the SubT Virtual Challenge.
Other
305 stars 98 forks source link

Camera data significantly delayed when using newest osrf/subt-virtual-testbed image #371

Closed osrf-migration closed 4 years ago

osrf-migration commented 4 years ago

Original report (archived issue) by Malcolm Stagg (Bitbucket: malcolmst7).

Using the current osrf/subt-virtual-testbed image, I’m seeing large delays with an increasing backlog of data for the front/image_raw camera topic. This affects both the urban and cave worlds.

Example repro steps:

ign launch -v 4 urban_circuit.ign robotName1:=X1 robotConfig1:=X1_SENSOR_CONFIG_4
roslaunch subt_example teleop.launch
rviz

(Can also replace urban_circuit.ign with cave_circuit.ign, or use another robot base such as X2)

Looking at the /X1/front/image_raw topic in rviz, initially it is fine, but after several seconds the data becomes increasingly delayed and no longer lines up with the current position of the robot.

Also, if I run my robot controller, I start to see error messages similar to the message below as it looks up a tf transform based on each image timestamp. This delay continues to increase, and the machine seems to run out of memory after a while:

[ERROR] [1585265942.360834143, 60.804000000]: Lookup would require extrapolation into the past.  Requested time 35.452000000 but the earliest data is at time 50.800000000, when looking up transform from frame [X2N1/base_link/camera_front] to frame [X2N1/map]

I’m not sure exactly when this started, but can confirm the image from 12/20 was not affected.

osrf-migration commented 4 years ago

Original comment by Reid Sawtell (Bitbucket: rsawtell).

We are seeing this issue as well.

osrf-migration commented 4 years ago

Original comment by Arthur Schang (Bitbucket: Arthur Schang).

I just ran two separate tests and I couldn't reproduce this issue in a 10 minute run or a 14 minute run. Here's the screenshot of header timestamps for the front/image_raw, ign topic -e -t /clock, and /clock from left to right:

Screenshot from 2020-03-30 19-53-19.png

I ran the exact setup you described above (even the rviz to view the image stream to potentially introduce lag) and I was unable to reproduce any impact on the header timestamps. There may be some delay on RViz being built up though, I did not experiment with that.

osrf-migration commented 4 years ago

Original comment by Malcolm Stagg (Bitbucket: malcolmst7).

Thanks Arthur for testing this. Was that with the new image from yesterday? I haven’t tried that one yet but will do that shortly. I saw the problem on the image from 03/23.

osrf-migration commented 4 years ago

Original comment by Malcolm Stagg (Bitbucket: malcolmst7).

This does appear to be fixed for me too in the image from 03/29. Maybe PR 429 had the side effect of fixing this too? Anyway, it looks good now. Thanks again for checking!

osrf-migration commented 4 years ago

Original comment by Arthur Schang (Bitbucket: Arthur Schang).

Yes, that was with the new image. I'm glad it's resolved.

osrf-migration commented 4 years ago

Original comment by Arthur Schang (Bitbucket: Arthur Schang).

Check two comments above, reopen if this reappears.