question about the scene rendered

[lihao0374]

Thanks for your wonderful work !!!

We evaluated the checkpoint you provided on the calvin benchmark. I found that the simulation scene rendered by the engine was not very clear and there was obvious noise, as shown in the attached image. What is the reason for this?

Looking forward to your reply. Thanks a lot !

[twke18]

Hi,

We believe it's the rendering issue on the simulator. I'll suggest rendering images with a higher resolution if you'd like a better visualization. But we follow the common setup using a resolution of 200x200 & 84x84 on Calvin.

Here is some example code we used internally for visualizing higher-resolution images:

import hydra

def get_env(dataset_path, obs_space=None, show_gui=True, **kwargs):
    from pathlib import Path

    from omegaconf import OmegaConf

    render_conf = OmegaConf.load(Path(dataset_path) / ".hydra" / "merged_config.yaml")

    if obs_space is not None:
        exclude_keys = set(render_conf.cameras.keys()) - {
            re.split("_", key)[1] for key in obs_space["rgb_obs"] + obs_space["depth_obs"]
        }
        for k in exclude_keys:
            del render_conf.cameras[k]
    if "scene" in kwargs:
        scene_cfg = OmegaConf.load(Path(calvin_env.__file__).parents[1] / "conf/scene" / f"{kwargs['scene']}.yaml")
        OmegaConf.update(render_conf, "scene", scene_cfg)
    if not hydra.core.global_hydra.GlobalHydra.instance().is_initialized():
        hydra.initialize(".")
    # These are two new lines which increase the resolution of the static camera
    render_conf['cameras']['static']['height'] = 512
    render_conf['cameras']['static']['width'] = 512
    env = hydra.utils.instantiate(render_conf.env, show_gui=show_gui, use_vr=False, use_scene_info=True)
    return env

def make_env(dataset_path, show_gui, split="validation", scene=None):
    val_folder = Path(dataset_path) / f"{split}"
    if scene is not None:
        env = get_env(val_folder, show_gui=show_gui, scene=scene)
    else:
        env = get_env(val_folder, show_gui=show_gui)

    return env

env = make_env(args.root_dir, False, args.split, f'calvin_scene_{args.scene}')

[lihao0374]

It seems that the GPU was being heavily utilized by other tasks, which consumed excessive computing resources and resulted in poor rendering quality. This even impacted the evaluation results of the policy. Therefore, we found an idle machine that was not being used. As a result, the rendering quality has significantly improved, and the evaluation results are now similar to those reported in the paper.