Open DavidPL1 opened 3 days ago
@DavidPL1 Thanks for the proposal. @Kallinteris-Andreas will know best, but I think this sounds like a reasonable idea.
Having a quick look at your implementation, currently you return a tuple of the rgb and the depth.
However as they have the same shape, it would be easier to concatenate the results as rgbd
of (480, 480, 4)
However as they have the same shape, it would be easier to concatenate the results as rgbd of (480, 480, 4)
Correct, however, the rgb image is of type np.uint8 while the depth image is of type np.float32. So this more of a design choice whether to have both images usable the same way as they are returned from the individual render modes or nicely fit them in a single array but require users to convert rgb back to uint8 from float32.
I am not familiar with how rgbd inputs are handled in CNNs, we should choose the output format for For whatever input is convenient for CNNs.
the rgb image is of type np.uint8 while the depth image is of type np.float32.
Ahh that makes sense, lets make a tuple like your implementation
@DavidPL1 could you make a PR with your branch?
Proposal
My proposal is to add a new
render_mode
to MuJoCo environments for when RGB and Depth images are required as observations, e.g. to create point clouds.(related issue: #727)
Motivation
Currently one can achieve this by calling
MujocoEnv.mujoco_renderer.render
twice with bothrender_mode=rgb_array
andrender_mode=depth_array
respectively. However, internally rgb and depth are always rendered and only one is returned based on therender_mode
. Therefore, offering to return both rgb and depth in one step saves the effort of rendering both images twice.Pitch
'rgbd_array' should be added as valid render_mode that returns both the rgb and the depth image either separately or as a 4 channel image (which entails conversion from uint8 to float32 and back for rgb).
Alternatives
No response
Additional context
I've already implemented a solution in my fork that returns both images separately.
Here's a comparison of execution times produced by the following script:
Results:
Checklist