Missing option to render RGB array with high-quality rendering

[jonasreiher]

Since version 3.0.1 you no longer pass a render argument during environment initialization but the render_mode argument instead. With this, the mode option of the environment's render() method has vanished.

I'm doing vision-based RL and require the following, all at the same time:

1. Full quality rendering as seen in human mode
2. Option to render to RGB arrays
3. Faster than real-time rendering (not limited to 25 fps)

Currently I can get either 1 or 2 and 3 but not all three. Am I missing something?

This was previously possible with gym.make(..., render=True) and env.render(mode='rgb_image'). I would suggest to keep the render method's mode option, removed in 3.0.1.

[qgallouedec]

This change is the consequence of gymnasium new rendering setting. See the gymnasium documentation.

I realise that I've made a mistake in rendering function (extra indentation causes rendering to return None when "render_mode"="human". I'm fixing this in #56. Could you try it?

When fixed, I would suggest to use

import gymnasium as gym
import panda_gym

env = gym.make('PandaReach-v3', render_mode="human")

observation, info = env.reset()

for _ in range(100):
    action = env.action_space.sample() # random action
    observation, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        observation, info = env.reset()
    rgba_array = env.render()

As pybullet is pretty slow for rendering, I'm not sure you'll reach high fps. You can speed up rendering by lowering the image size. Your best option is t modify the source code of panda-gym (change the defaults value for rendering, currently 720x480). In the futur, it could be great to have parameters like render_width and render_height in the env arguments.

[jonasreiher]

Continued discussion in #56.

Regarding modifying the image size: Instead of modifying the source code and calling env.render() it's possible to call env.sim.render(height=..., width=...). This of course won't work with wrappers calling env.render().

Side note regarding rendering speed: Is human rendering expected to be faster than the rgb_array mode thanks to OpenGL? I'm seeing 40% slower rendering times for the latter.

[qgallouedec]

I'm not sure how to handle that. The gym doesn't seem to be consistent in this respect. See https://github.com/Farama-Foundation/Gymnasium/issues/283

[jonasreiher]

This was more of a note than an implementation request. In panda-gym, the render method of env.sim does support height/width, while the one of env doesn't. This used to be different in 3.0.0, though.

[araffin]

Related: https://github.com/Farama-Foundation/Gymnasium/issues/100 (SB3 uses rgb_array and OpenCV to be able to also display a "human" render)