search

google / brax

Massively parallel rigidbody physics simulation on accelerator hardware.
Apache License 2.0
2.34k stars 255 forks source link

placing camera in simulation world #26

Closed flobotics closed 3 years ago

flobotics commented 3 years ago

hi, is there a way to place a camera and then use this camera image inside the step() and _get_obs() methods ?

cdfreeman-google commented 3 years ago

Hiya!

We don't currently support visual observations, but this is on our todo-list, as it's a fairly common ask. Will update this issue once in lands, probably within the next month.

flobotics commented 3 years ago

sounds good

flobotics commented 3 years ago

if you want to test the trained model in the real world, its very hard to setup the realworld-camera in the same angle/position as the camera in the simulation.

It would be nice if the user could e.g. define a sphere/box around the simulated-camera-placement and in training the simulated-camera-position changes slightly inside the defined sphere/box. So the model will get more generic.

Perhaps the user need also to define a e.g. arrow that points from the sphere/box to the point-of-interrest and then the camera-position can change but it still knows where to point to the camera.

Another thing with camera too would be to add a plane where pictures can be displayed. These planes could then be used as ground or walls, so that in every new training episode another picture could be displayed and the camera can do Domain-randomization, also nice for generic models.

mweiss17 commented 3 years ago

Hello, Just posting to mention that I'm also interested in having visual observations -- ideally rgbd -- thanks for your work! Martin

fgolemo commented 3 years ago

Same here. This feature would be great! Also great job folks!

erwincoumans commented 3 years ago

There is a slow cpu camera implementation that you could try out, based on PyTinyRenderer. This is the same renderer using in PyBullet, when no GPU OpenGL/EGL hardware is available.

PyTinyRenderer can render RBG, depth and segmentation mask. It supports cubes, capsules and general triangle meshes. Texture mapping and shadow map. The renderer is not anywhere close to Unity or Unreal but maybe sufficient for some learning projects.

See https://github.com/google/brax/blob/main/brax/io/image.py and this colab, scroll down to the render image function.

a colab how to use pytinyrenderer standalone.

Closing for now. If this doesn't work for you, please re-open.