search

google-deepmind / mujoco

Multi-Joint dynamics with Contact. A general purpose physics simulator.
https://mujoco.org
Apache License 2.0
8.19k stars 818 forks source link

[feature request] fisheye camera + realtime PBR #1859

Closed ykim104 closed 3 months ago

ykim104 commented 3 months ago

Is your feature request related to a problem? Please describe. We are using fisheye cameras as visual sensors to teach a robot policy and would like to use Mujoco for generating robot trajectories with visual observations.

Describe the solution you'd like A fisheye tag for creating a body/camera

Describe alternatives you've considered As an alternative, we would need to try to expand Mujoco's default OpenGL renderer to create a fisheye effect ourselves

Additional context

aftersomemath commented 3 months ago

MuJoCo uses the fixed function OpenGL pipeline. I'm not certain, but I don't think it can be used to render distorted images. Usually a shader would be used for this.

However, it is straightforward to render several regular pinhole cameras and create a fisheye image by sampling from the pinhole camera frames.

ykim104 commented 3 months ago

Thank you @aftersomemath for your reply!

I have another feature request/question. Is there a plan to bring in other OpenGL features like physically-based rendering (PBR) or make use of other textures like Normal maps? We believe such improvements in render fidelity could really help the research. Thank you in advance for your feedback!!

yuvaltassa commented 3 months ago

Yes, we are planning some significant rendering improvements, but these will take some time (end-of-year-ish). You can already see some early aspects of this as materials can now reference various kinds of textures, but these are currently ignored by the native renderer (except for RGB). Closing for now, the rendering improvements, when they are done, will be well-advertised :slightly_smiling_face: