RGB camera with `render_pipeline` has all black sky

[pengzhenghao]

Setting:

1. image_observation = False
2. use_render = True
3. sensors=dict(rgb_camera=(RGBCamera, 512, 256))

Image is collected via:

img_obs = ImageObservation(cfg, image_source="rgb_camera", clip_rgb=False) img_obs.observe()

[QuanyiLi]

When using the render pipeline, only the main camera can be used. The wrong rendering results for other sensors are as expected.

[pengzhenghao]

So do we have any way to generate first-view cam images with deluxe RGB?

[QuanyiLi]

sensors=dict(main_camera=()),
camera_dist=-1,
camera_pitch=15,
camera_height=1,

[pengzhenghao]

The config does not work for headless setting. It works for head machine:

When running the following script:

from metadrive.envs import MetaDriveEnv
import matplotlib.pyplot as plt
import os

size = (256, 128) if not os.getenv('TEST_DOC') else (16, 16)  # for github CI

env = MetaDriveEnv(dict(
    # log_level=50,  # suppress log
    image_observation=True,
    # show_terrain=not os.getenv('TEST_DOC'),

    render_pipeline=True,  # Turn on deluxe rendering
    sensors=dict(main_camera=()),
    camera_dist=-1,
    camera_pitch=15,
    camera_height=1,

    vehicle_config={"image_source": "main_camera"},
    # stack_size=3,
))
obs, info = env.reset()
for _ in range(5):
    obs, r, d, t, i = env.step((0, 1))

env.close()

print({k: v.shape for k, v in obs.items()})  # Image is in shape (H, W, C, num_stacks)

# Reverse channel:
obs["image"] = obs["image"][..., ::-1, :]

plt.subplot(131)
plt.imshow(obs["image"][:, :, :, 0])
plt.subplot(132)
plt.imshow(obs["image"][:, :, :, 1])
plt.subplot(133)
plt.imshow(obs["image"][:, :, :, 2])

plt.show()

The error:

[INFO] Environment: MetaDriveEnv
[INFO] MetaDrive version: 0.4.2.3
[INFO] Sensors: [lidar: Lidar(), side_detector: SideDetector(), lane_line_detector: LaneLineDetector(), main_camera: MainCamera(1200, 900)]
[INFO] Render Mode: offscreen
[INFO] Horizon (Max steps per agent): None
[INFO] Assets version: 0.4.2.3
[INFO] Known Pipes: glxGraphicsPipe
[INFO] Assets version: 0.4.2.3
[INFO] Known Pipes: glxGraphicsPipe
Assertion failed: _cache_ref_count > 0 at line 190 of built1.10/include/cachedTypedWritableReferenceCount.I
Assertion failed: _node_ref_count == 0 at line 93 of built1.10/include/nodeCachedReferenceCount.I
Traceback (most recent call last):
  File "/home/zhenghao/anaconda3/envs/infgen/lib/python3.10/site-packages/direct/showbase/ShowBase.py", line 2145, in __garbageCollectStates
    RenderState.garbageCollect()
AssertionError: _cache_ref_count > 0 at line 190 of built1.10/include/cachedTypedWritableReferenceCount.I
Traceback (most recent call last):
  File "/home/zhenghao/metadrive/tmp.py", line 21, in <module>
    obs, info = env.reset()
  File "/home/zhenghao/metadrive/metadrive/envs/base_env.py", line 518, in reset
    self.lazy_init()  # it only works the first time when reset() is called to avoid the error when render
  File "/home/zhenghao/metadrive/metadrive/envs/base_env.py", line 410, in lazy_init
    initialize_engine(self.config)
  File "/home/zhenghao/metadrive/metadrive/engine/engine_utils.py", line 38, in initialize_engine
    cls.singleton = cls(env_global_config)
  File "/home/zhenghao/metadrive/metadrive/engine/base_engine.py", line 88, in __init__
    self.warmup()
  File "/home/zhenghao/metadrive/metadrive/engine/base_engine.py", line 751, in warmup
    self.taskMgr.step()
  File "/home/zhenghao/anaconda3/envs/infgen/lib/python3.10/site-packages/direct/task/Task.py", line 504, in step
    self.mgr.poll()
  File "/home/zhenghao/anaconda3/envs/infgen/lib/python3.10/site-packages/direct/showbase/ShowBase.py", line 2145, in __garbageCollectStates
    RenderState.garbageCollect()
AssertionError: _cache_ref_count > 0 at line 190 of built1.10/include/cachedTypedWritableReferenceCount.I

Process finished with exit code 1