No response when recording a video

[huiwenzhang]

Basically, I just load a xml from the path, create the model, simulate it and render the scene. Example code is as follows:

model = load_model_from_path(xml_path)
sim = MjSim(model)
viewer = MjViewer(sim)

t = 0
if t < 1000:
    sim.step()
    viewer.render()
    t += 1

It works fine. But once I try to record the video by turning on the record video option (click V button), the window turns dark and got no response.

[marjanin]

I am having the same issue on Mac

[Kelym]

+1 with same problem on Ubuntu 16.04

[John-HarringtonNZ]

Same issue as well. I have spent some time looking into the reason for this, and it seems to be related to how Mac works with GLFW. It occurs when the the _read_pixels_as_in_window function is called here.

Particularly when the offscreen context is rendered. I'm not familiar enough with glfw yet, but I'm looking into it now. If anyone has any ideas that would be great.

I've got the same issue on both Ubuntu 20.04 as well as Mac Big Sur, but with different outputs. Ubuntu just changes the render resolution, while Mac also messes up the viewer orientation/position.

[stefanwanckel]

Is there any update on this or maybe some alternative to record the simulation window?

[LucMc]

I developed a script using open-cv to record its rendering screen instead. It's not a perfect solution but is nonetheless effective. In the example below I am loading a stable_baselines model for testing.

import gym
from stable_baselines3 import SAC, PPO
import matplotlib.pyplot as plt
from matplotlib import animation
import cv2
import numpy as np
import pyautogui
import time

ENV_ID = "FetchReach-v1"
REPLAY_PATH = "./videos/"
ALGO = "SAC"
HER = True

def save_frames_as_gif(frames, path='./', filename='operator.gif'):
    frames = frames[2:]
    plt.figure(figsize=(frames[0].shape[1] / 72.0, frames[0].shape[0] / 72.0), dpi=72)
    patch = plt.imshow(frames[0])
    plt.axis('off')

    def animate(i):
        patch.set_data(frames[i])

    anim = animation.FuncAnimation(plt.gcf(), animate, frames = len(frames), interval=50)
    anim.save(path + filename, fps=20)

model_path = f"./models/{ENV_ID}/local/{ALGO}/normal{'_HER' if HER else ''}"

RES = (70,70,1800,1130)
pyautogui.moveTo((7,1125))
env = gym.make(ENV_ID)
if ALGO == "SAC":
    model = SAC.load(model_path, env=env)
elif ALGO == "PPO":
    model = PPO.load(model_path, env=env)

obs = env.reset()
done = False
frames = []

while not done:
    env.render()
    time.sleep(0.001)
    action, _ = model.predict(obs, deterministic=True)
    # action = env.action_space.sample()
    img = pyautogui.screenshot(region=RES)
    frame = np.array(img)
    frames.append(frame)

    obs, reward, done, info = env.step(action)

save_frames_as_gif(frames, path=REPLAY_PATH, filename=ENV_ID+".gif")