search

shacklettbp / madrona

MIT License
305 stars 30 forks source link

roadmap? #34

Closed yangfengzzz closed 3 months ago

yangfengzzz commented 3 months ago

This is a very interesting project. Since 2021, there has been a series of work on GPU simulation and GPU Batch rendering, but this project shows the possibility of a one-stop solution. I would like to know if there is a roadmap for this project? I noticed that there is another project trying to integrate MJX. Will this project use Jax as the scripting language and use MJX as the default physics engine in the future?

shacklettbp commented 3 months ago

Thanks for your interest! We don't have a public roadmap, unfortunately. Features we implement are currently motivated by our own research needs and those of our collaborators. Feel free to reach out with specifics (including directly to my email), if there are missing features that you would need for a potential project!

The madrona_mjx repo just integrates Madrona as a batch renderer for MJX. Unfortunately the static shaping constraints of MJX and jax in general are not a great fit for a Madrona scripting language (Madrona supports dynamic numbers of entities per environment).

yangfengzzz commented 3 months ago

Thanks for your reply. MJX does lack a usable renderer. The currently reused Mujoco renderer will cause a lot of copies between CPU and GPU, so I had a similar idea before. In addition, I would like to ask whether the renderer of the bps3D project has been integrated into Madrona?

shacklettbp commented 3 months ago

No, Madrona has a two new renderer backends, one based on the HW rasterizer and one based on software raycasting. The software raycasting is likely better for most deep RL applications. The rendering is currently basic, but if you want to do depth rendering or very simple RGB Madrona likely already has what you need.

yangfengzzz commented 3 months ago

Does software raycasting only generate depth maps or does it also include RGB? I noticed that IsaacSim can provide very realistic PBR rendering effects. I wonder if this is the future development direction of madrona? I rarely do visual RL, so I don’t know whether the realism of vision will be of great help to RL convergence.

shacklettbp commented 3 months ago

The raycasting backend can do simple RGB. Increasing the visual fidelity of the RGB rendering is not on the roadmap currently.

yangfengzzz commented 3 months ago

THX