Glitches in video creation

[jenhicks]

As of OpenSim 4.0, when playing back motions, the video recorded can be "jumpy" (e.g., when playing back at slow speeds). Would be helpful to be able to set a certain frame rate, playback speed, length for creating a video and not worry about how it plays back on the screen. A work around in the meantime could be to output still frames as we did in previous versions.

[chrisdembia]

I know this isn't the best place to provide this comment, but by far my biggest pain point with the GUI now is creating and saving movies. My requests are:

• Saving the movie to MP4 format.
• Recording starts and stops when the motion starts and stops, not when the user clicks record (or for this to at least be a default mode).
• Smoother playback (same as this issue).

[jenhicks]

Thanks for reporting and sorry for the pain! @aymanhab Would upgrading help with any of these issues?

[aymanhab]

Thanks for the feedback @chrisdembia

1. Not sure if saving into MP4 is now built into a library that we can use, worth revisiting indeed.
2. It is straightforward to start recording on first animation frame, but having two buttons (play motion and record video) tied together can have other side effects, will try to come up with a reasonable proposal,
3. Smoother playback and claims about frame-rate are all due to users putting too many models/objects increasing frame rendering time. Used simple pendulum model to prove this to be the issue. If rendering time is indeed the problem then only solution is offline rendering which is a separate major undertaking.

[carmichaelong]

I dove into this a little more, and am happy to check some other cases too if they're needed to help isolate the problem or test more possible common use cases (I don't think what is below is exhaustive by any means).

I think there are two cases with reasonable expectations:

1. Recording videos should always have smooth frame rates in the output. I also think that, as a user, it's reasonable for this to take longer than real time to create. To test this, I made 3 videos with my laptop using the ToyDropLanding model shipped with OpenSim and creating a 2 second forward simulation. For each video, I synced between 1, 2, or 3 models doing the same motion. In all output videos, the frame rate was inconsistent, and worsens with more models. These can be found in the .zip attached below. drop_models.zip

2. [This second case may need its only issue] Visualizer should be smooth when doing reasonable playback/dragging time real time in the visualizer for a single model with limited data. I tested this using a very powerful desktop. The .gif's below show me loading the 2392 model and a short .mot file provided in the OpenSim distribution. In the first case I am slowly dragging time forward, and in the second case, I am going both forward and backward. From conversation with others in the past, it seems like normal playback on slower machines could also cause this to happen.

I think there are also other logistical issues brought up by @chrisdembia from before that I agree with after working on this issue more along with some other small issues.

1. .mp4 (or .avi) would be way better than .webm
2. Syncing with motion starting and stopping would be great. Perhaps this could also be easier if a video is not created in real time.
3. Unsure why this happens, but when I play the .webm file with a native Windows app, the video does not allow me to change time on the fly (I can only ever play or pause the video, not drag the time to a specific spot). It looks like the player also doesn't know how long the video is, which may be related.

[aymanhab]

Thanks for the investigation @carmichaelong definitely very helpful. I'd also suggest you try simple model (e.g. DoublePendulum, bouncing_block or Leg39) and do some subset of the scenarios and see if the issues do reproduce. My conjecture is that all these issues are due to rendering taking too long so we're dealing with workarounds to "challenged" hardware, but I could be wrong. Note that a powerful machine or graphics card may not be the only factor if "Chrome" is running and using 20 tabs that are killing the hardware/gfx card 😃

[carmichaelong]

Thanks for the feedback. As a note, for the tests in part 2 it was run clean without other programs using the CPU and GPU (and I do think that my setup should be considered more powerful than what we expect our users to have too). For part 1, I think it's important for any user to have consistent video frames regardless of setup (e.g., in cases where part 2 doesn't work smoothly on their setup), even if it takes a long time.

I agree with you that the model complexity affects the performance of the visualizer, and that should be expected. I chose the cases above because I do believe they are a good representation of the typical cases for our users, as they are two models that are used often in our examples and tutorials. Because of that, I'd like to at least keep those as part of the benchmark, and we can add other cases as needed. The three cases you listed could be added, but I don't think that they alone span typical use cases. I am happy to continue to help test all important cases across different machines and give feedback.

The visualizer is one of the most important parts of OpenSim, and the new visualizer you brought to 4.0 has already been a great upgrade to creating high-quality images as seen in the book and paper. Making sure that all our users can also easily share their cool work through high-quality videos would be an awesome way to increase OpenSim's impact and reach even more users.

[aymanhab]

For the recorded/zipped videos, checking "Properties" of all these files show frame rate of 30 fps

[jenhicks]

I believe our response to this one was to allow the generation and saving of "stills". Where is this documented so I can try it out? @aymanhab Thanks!

[aymanhab]

Sorry @jenhicks here's the PR https://github.com/opensim-org/opensim-gui/pull/1234 There's a line in the changelog file of the GUI about change video format but I don't think it's clear. Let me know if you have ideas to make this more accessible. Thank you

[jenhicks]

@aymanhab Please add instructions about how to do this here: https://simtk-confluence.stanford.edu/display/OpenSim/Taking+Snapshots+and+Making+Movies#TakingSnapshotsandMakingMovies-RecordingaMoviewRecordingaMovie