Understanding real time and video rendering

[eferreirafilho]

Hi all, since the commit that added "add viewport recording feature", I'm using this to record videos of the training and inference. However, I'm having a hard time understanding the time scales. I have implemented a simulation using the FrankaDeformable as a basis.

In config.yaml, I have:

# disables rendering
headless: False
# enables native livestream
enable_livestream: False
# timeout for MT script
mt_timeout: 300

# enables viewport recording
enable_recording: True
# interval between video recordings (in steps)
recording_interval: 5000
# length of the recorded video (in steps)
recording_length: 900
# fps for writing recorded video
recording_fps: 60
# directory to save recordings in
recording_dir: ''

in the cfg/task file, I have:

sim:
  dt: 0.0041 #1/240 s #0.0083 # 1/120 s
  use_gpu_pipeline: ${eq:${...pipeline},"gpu"}
  gravity: [0.0, 0.0, -9.81]
  add_ground_plane: True
  add_distant_light: False
  use_fabric: True
  enable_scene_query_support: False
  disable_contact_processing: False

When I run inference with headless=False, I can see the viewport with a varying FPS that stays close to 20 fps.

This way I can create 30 seconds videos (attached), but, how they relate to real time?

https://github.com/NVIDIA-Omniverse/OmniIsaacGymEnvs/assets/92797165/53a3e0d3-407e-47c3-87d6-46fe73808903

To test this, I have set an object at 10 meters high, it should take 1.414 seconds to fall to the ground. But in the rendered video, it only drops around the 10-second mark. This isn't right. What I'm missing?

More importantly, how can I render videos that are real time?

Thanks in advance!

[tolomeis]

Hi, I've been messing around with video recording in this framework for some time, enough to understand what's happening and how to record videos in real-time.

TL;DR:

Video recording using enable_recording: True is always at 30 sim steps per second.

To solve all of this and get real-time videos, I post-process the videos with ffmpeg, using the simulation rate of my setup. In my case I have dt: 0.005 so a rate of 200Hz:

ffmpeg -i rl-video-step-0.mp4 -vf "setpts=30/200*PTS" -c:v libx264 -c:a copy rl-video-step-0-REALTIME.mp4

This will speed up the video in order for it to be real-time.

What's happening

OmniIsaacGymEnvs uses an old version of OpenAI Gym (23.0.1) which has this bug that overwrites user recording_fps to 30. Basically, the recording using the enable_recording: True happens always at 30 frames / second, with each frame being a simulation step of Isaac Sim. In your case you have dt: 0.0041 which means that it takes 345 simulation steps to make 1.414 seconds. When recorded at 30fps that translates to 11.5 seconds, which is approximately when we can see the object falling to the ground in your video.

Since this is happening inside OpenAI Gym, the recording_fps parameter does absolutely nothing to changes this, and all it does is re-encode the video to match the desired frame rate, without actually changing the run time of the video.

This problem was fixed here and I think was included in 0.24.0, which unfortunately brings breaking changes to the API, so it won't work immediately with OmniIsaacGymEnvs, but at least it would be nice to see a warning on the known issues.

Hope this helps, Simone

[eferreirafilho]

That's super helpful information! Thanks!