search

osrf / subt

This repostory contains software for the virtual track of the DARPA SubT Challenge. Within this repository you will find Gazebo simulation assets, ROS interfaces, support scripts and plugins, and documentation needed to compete in the SubT Virtual Challenge.
Other
309 stars 97 forks source link

Camera Rendering Broken on AWS #635

Closed rsawtell closed 4 years ago

rsawtell commented 4 years ago

Images we bagged from our latest run reveal some sort of camera simulation failure on the AWS, namely that all or part of the world appears to disappear from the perspective of the camera system leaving a mostly black screen in which only certain things (artifacts and other robots) appear to be rendered. The failure seems intermittent, but affected every robot at some point.

Valid image taken by a UAV, and a corrupted image a few seconds later: Xd_sidebyside

Same UAV later in the simulation, part of the world is visible while other parts are not: Xe_2020273090909041170

View of a rescue randy by a UGV, note the shadowed area at the bottom should be obscured by terrain (you may have to adjust your contrast but you can see the entire rescue randy, it's just darker): Xb_xray

Not sure if this is related or not, but the log playback seems mostly fine except depending on the perspective you get things like this (the camera is not inside any geometry): image

or as a GIF: PlaybackWeirdness

For reference the simulation id is 165fd66a-ab7f-471e-91b7-a49cf616f129

nkoenig commented 4 years ago

Thanks. I'm looking into it.

iche033 commented 4 years ago

@rsawtell, are the images from an rgbd camera? Do you know the approximate locations in the world where the UAV captured these images?

rsawtell commented 4 years ago

These are from HD camera configurations. The locations appear to be almost anywhere in the tunnel, it's not really limited to specific locations. In the above examples they are the helmet to the right after entering the tunnel, the starting area, and the first rescue randy on the left branch of the tunnel. Probably 90% of the images I captured had the problem though from basically anywhere the robots visited, and this seems new to this run as I've been capturing images for awhile now. I also don't see it in local development testing, though i haven't tried cloudsim.

peci1 commented 4 years ago

Just to make sure - do you use an english locale in your system? Try running the GUI with environment LC_ALL=C LANG=C.

rsawtell commented 4 years ago

We are running on an english locale.

We also saw this problem on our latest run: Run ID: b964c8b7-b8a5-42c1-9f31-b1368ed2be60

Xh_2020274110119642831

nkoenig commented 4 years ago

The issue is caused by surpassing a shadow caster limit in Ogre2. A patch is in progress. I the mean time you have a couple options:

  1. Run with fewer robots. I realize this is not a long term solution.
  2. Turn off shadow casting on some of the lights on your robots. This solution will only affect local runs. Look for the <cast_shadows> tag in the model.sdf file.

I'll update this issue when the patch is in place.

rsawtell commented 4 years ago

Thanks for the update, we'll keep this in mind.

malcolmst commented 4 years ago

For robot budgeting purposes, is it safe to assume this will be fixed before cave circuit?

nkoenig commented 4 years ago

Fix has been on Cloudsim, public dockerhub images are uploading. Release notes are here: https://github.com/osrf/subt/wiki/release_notes#2020-10-07,

malcolmst commented 4 years ago

Awesome! Thanks for the update.