[Bug Report] Bug in rendering camera images in Gym environment

[Aqua12138]

Describe the bug

• I'm using camera in a reinforcement learning framework and when I set headless to True, the program stops running

  Steps to reproduce

  The main bug

• I checked the code and it seems to be caused by the experience parameter of simulation_app, you can reproduce my error like this
• Open the ”Orbit/source/standalone/demo/play_camera.py“
• Replace the following code (because the RL fram use this code)

  simulation_app = SimulationApp(config)

• with

  import os
  config = {"headless": args_cli.headless}
  # load cheaper kit config in headless
  if args_cli.headless:
  app_experience = f"{os.environ['EXP_PATH']}/omni.isaac.sim.python.gym.headless.kit"
  else:
  app_experience = f"{os.environ['EXP_PATH']}/omni.isaac.sim.python.kit"
  # launch the simulator
  simulation_app = SimulationApp(config, experience=app_experience)

  • Commit: none
  • Isaac Sim Version: 2022.2.0
  • OS: Ubuntu 20.04
  • GPU: RTX 3090ti
  • CUDA: 11.7
  • GPU Driver: 515.86.01

  Additional context

  Add any other context about the problem here.

  Checklist

  • [ ] I have checked that there is no similar issue in the repo (required)
  • [ ] I have checked that the issue is not in running Isaac Sim itself and is related to the repo

[Mayankm96]

Are you running this in headless mode?

If so, the above changes will try loading the configuration file for Isaac Sim called "omni.isaac.sim.python.gym.headless.kit". This configuration file does not have the viewport extension enabled by default, which is needed for rendering camera images. We enable it explicitly in the RL environment if running in non-headless mode:

https://github.com/NVIDIA-Omniverse/Orbit/blob/main/source/extensions/omni.isaac.orbit_envs/omni/isaac/orbit_envs/isaac_env.py#L413

You can add these lines right after launching the simulator. They'll enable the extension manually.

from omni.isaac.core.utils.extensions import enable_extension

# enable viewport extension if not running in headless mode
enable_extension("omni.kit.viewport.bundle")

I think it would make sense to enable this explicitly in the camera class to avoid unnecessary duplication of this code at multiple places.

Let me know if the issue gets fixed with it.

[Aqua12138]

Yes, I want to run the camera in headless mode to collect data. Here are some of my attempts and results

• I modified the train code as you said, but I still get the same error

  
  # launch the simulator
  config = {"headless": args_cli.headless}
  # load cheaper kit config in headless
  if args_cli.headless:
  app_experience = f"{os.environ['EXP_PATH']}/omni.isaac.sim.python.gym.headless.kit"
  else:
  app_experience = f"{os.environ['EXP_PATH']}/omni.isaac.sim.python.kit"
  # launch the simulator
  simulation_app = SimulationApp(config, experience=app_experience)

revision

from omni.isaac.core.utils.extensions import enable_extension

enable viewport extension if not running in headless mode

enable_extension("omni.kit.viewport.bundle")

- I tried to stop loading `"omni.isaac.sim.python.gym.headless.kit"` and the good news is that the code works fine, but I checked the image data I got and I found that none of them changed, but in non-headless mode I can get several frames of changing image data. Is this because I didn't load `"omni.isaac.sim.python.gym.headless.kit"`?

Actually I just want to get the camera data in headless mode, because my environment has very low frame rate in non-headless mode, I hope to provide a demo of using the camera to collect data in headless mode, thanks a lot.

[Mayankm96]

Is it just with play_camera.py? I have a unit test that runs in headless mode and after a few initial blank frames (a known issue in the simulator), it renders fine. You can run the test script here for it:

./orbit.sh -p source/extensions/omni.isaac.orbit/test/test_camera.py

[Aqua12138]

Thanks, I ran test_camera.py and the camera still collects data normally in headless mode, but not in RL frame. I found the reason, it's because sim.step(render=False) is not collecting camera data, just set render=True to solve the problem. Thank you for your answer.