[Question] How to use the vision-based Safety Gymnasium in a headless server

[hdadong]

Hi, I've recently come across the following error, on a machine with Nvidia driver version: 515.76, CUDA Version: 11.7, when trying to use the vision-based Safety Gymnasium in a headless Ubuntu 20.04 remote server. The same exact code was running properly on a machine with display.

Any idea how to fix the issue?

The program I run is as follows:

import argparse
import os

# import gymnasium
import safety_gymnasium
from gymnasium.utils.save_video import save_video

WORKDIR = os.path.abspath('.')
DIR = os.path.join(WORKDIR, 'omnisafe/envs/safety-gymnasium/examples', 'cached_test_vision_video')

def run_random(env_name):
    env = safety_gymnasium.make(env_name)
    # env.seed(0)
    obs, _ = env.reset()
    terminled = False
    ep_ret = 0
    ep_cost = 0
    render_list = []
    for i in range(1001):
        if terminled:
            print('Episode Return: %.3f \t Episode Cost: %.3f' % (ep_ret, ep_cost))
            ep_ret, ep_cost = 0, 0
            obs, _ = env.reset()
            save_video(
                frames=render_list,
                video_folder=DIR,
                name_prefix=f'test_vision_output',
                fps=30,
            )
            render_list = []
        assert env.observation_space.contains(obs)
        act = env.action_space.sample()
        assert env.action_space.contains(act)
        # Use the environment's built_in max_episode_steps
        if hasattr(env, '_max_episode_steps'):
            max_ep_len = env._max_episode_steps
        render_list.append(obs['vision'])
        obs, reward, cost, terminled, truncated, info = env.step(act)

        ep_ret += reward
        ep_cost += cost

if __name__ == '__main__':

    parser = argparse.ArgumentParser()
    parser.add_argument('--env', default='SafetyCarGoal0Vision-v0')
    args = parser.parse_args()
    run_random(args.env)

which produced errors:

(mbppo) $ python safety_gym_v2_vision.py 
/home/weidong/omnisafe/omnisafe/envs/Safety_Gymnasium/safety_gymnasium/utils/passive_env_checker.py:49: UserWarning: WARN: A Box observation space has an unconventional shape (neither an image, nor a 1D vector). We recommend flattening the observation to have only a 1D vector or use a custom policy to properly process the data. Actual observation shape: (3, 3)
  logger.warn(
/home/weidong/anaconda3/envs/mbppo/lib/python3.8/site-packages/glfw/__init__.py:912: GLFWError: (65544) b'X11: The DISPLAY environment variable is missing'
  warnings.warn(message, GLFWError)
/home/weidong/anaconda3/envs/mbppo/lib/python3.8/site-packages/glfw/__init__.py:912: GLFWError: (65537) b'The GLFW library is not initialized'
  warnings.warn(message, GLFWError)
Traceback (most recent call last):
  File "safety_gym_v2_vision.py", line 66, in <module>
    run_random(args.env)
  File "safety_gym_v2_vision.py", line 31, in run_random
    obs, _ = env.reset()
  File "/home/weidong/omnisafe/omnisafe/envs/Safety_Gymnasium/safety_gymnasium/wrappers/order_enforcing.py", line 57, in reset
    return self.env.reset(**kwargs)
  File "/home/weidong/omnisafe/omnisafe/envs/Safety_Gymnasium/safety_gymnasium/wrappers/env_checker.py", line 60, in reset
    return env_reset_passive_checker(self.env, **kwargs)
  File "/home/weidong/omnisafe/omnisafe/envs/Safety_Gymnasium/safety_gymnasium/utils/passive_env_checker.py", line 214, in env_reset_passive_checker
    result = env.reset(**kwargs)
  File "/home/weidong/omnisafe/omnisafe/envs/Safety_Gymnasium/safety_gymnasium/envs/safety_gym_v2/builder.py", line 183, in reset
    return (self.task.obs(), info)
  File "/home/weidong/omnisafe/omnisafe/envs/Safety_Gymnasium/safety_gymnasium/envs/safety_gym_v2/tasks/goal/goal_level0.py", line 214, in obs
    obs['vision'] = self.obs_vision()
  File "/home/weidong/omnisafe/omnisafe/envs/Safety_Gymnasium/safety_gymnasium/envs/safety_gym_v2/base_task.py", line 375, in obs_vision
    vision = self.engine.render(width, height, mode='rgb_array', camera_name='vision', cost={})
  File "/home/weidong/omnisafe/omnisafe/envs/Safety_Gymnasium/safety_gymnasium/envs/safety_gym_v2/engine.py", line 339, in render
    self._get_viewer(mode).render(camera_id=camera_id)
  File "/home/weidong/omnisafe/omnisafe/envs/Safety_Gymnasium/safety_gymnasium/envs/safety_gym_v2/engine.py", line 453, in _get_viewer
    self.viewer = RenderContextOffscreen(self.model, self.data)
  File "/home/weidong/anaconda3/envs/mbppo/lib/python3.8/site-packages/gymnasium/envs/mujoco/mujoco_rendering.py", line 232, in __init__
    super().__init__(model, data, offscreen=True)
  File "/home/weidong/anaconda3/envs/mbppo/lib/python3.8/site-packages/gymnasium/envs/mujoco/mujoco_rendering.py", line 57, in __init__
    self.con = mujoco.MjrContext(self.model, mujoco.mjtFontScale.mjFONTSCALE_150)
mujoco.FatalError: gladLoadGL error

[zmsn-2077]

For those who are using a headless linux box without X-display, use this:

export MUJOCO_GL=osmesa
export MJLIB_PATH=$HOME/.mujoco/mujoco200/bin/libmujoco200.so
export MJKEY_PATH=$HOME/.mujoco/mujoco200/mjkey.txt
export LD_LIBRARY_PATH=$HOME/.mujoco/mujoco200/bin:$LD_LIBRARY_PATH
export MUJOCO_PY_MJPRO_PATH=$HOME/.mujoco/mujoco200/
export MUJOCO_PY_MJKEY_PATH=$HOME/.mujoco/mujoco200/mjkey.txt

For more details, refer to denisyarats/dmc2gym#4

[friedmainfunction]

For those who are using a headless linux box without X-display, use this:

export MUJOCO_GL=osmesa
export MJLIB_PATH=$HOME/.mujoco/mujoco200/bin/libmujoco200.so
export MJKEY_PATH=$HOME/.mujoco/mujoco200/mjkey.txt
export LD_LIBRARY_PATH=$HOME/.mujoco/mujoco200/bin:$LD_LIBRARY_PATH
export MUJOCO_PY_MJPRO_PATH=$HOME/.mujoco/mujoco200/
export MUJOCO_PY_MJKEY_PATH=$HOME/.mujoco/mujoco200/mjkey.txt

For more details, refer to denisyarats/dmc2gym#4

Due to mujoco now is installed by pip, so we just need to add first line into ~/.bashrc. and you should refresh it to activate modifications.

source ~/.bashrc