GStreamer stuttering on Windows (resolved by disabling GL path)

[benjaminbojko]

I've been running some tests with Gstreamer and am excited for it to replace QuickTime at some point hopefully. Without any modifications, I'm getting heavy stuttering/jumping back-and-forth across frames, though.

Following the instructions @PetrosKataras posted in the following PR, I was able to get smooth playback by disabling gl path: https://github.com/cinder/Cinder/pull/2044#issuecomment-434586119

My other settings were:

• Windowed mode (fullscreen on and after launch tested too, no difference)
• Vsync enabled
• Framerate disabled

All of this is running off a MacBook Pro with an i7 and Radon Pro 560 running the latest Windows 10. I have gstreamer packages v 1.12.4 installed as recommended here: https://gstreamer.freedesktop.org/data/pkg/windows/1.12.4/gstreamer-1.0-devel-x86_64-1.12.4.msi

My question: Should that option be disabled by default for Gstreamer? Is there a benefit to keeping it enabled? Would upgrading gstreamer help?

[richardeakin]

I think I saw this one time, but for whatever reason the next time I tested I couldn't reproduce. Sorry that doesn't help much, just sharing what I observed a while back.

[PetrosKataras]

Not sure what exactly is happening here. I cannot reproduce this on my test setup with Nvidia hardware. I would guess that this is somehow related to different drivers in this case?

@benjaminbojko Are you are running this inside a Windows VM on a macOS host? Or are you using something like Boot Camp for this? I m wondering if this is also somehow related ( especially if you are running inside a VM ). Would it be possible to give it a try on another Windows machine and see if you get the same results?

This specific option is there because it makes video playback more performant, which for a few videos or/and lower resolutions won't be that noticeable since GStreamer has optimized its traditional CPU path in later versions but as you scale up in number of videos and/or resolutions, then it can really make a difference. That said, maybe switching the default to CPU and providing a configure flag for enabling the GL path when there is an identified bottleneck on video playback might be a better ( more widely supported ) default option.

As for updating to the latest version of GStreamer, personally I would always recommend to keep up to date with the latest ( and 1.14 has brought quite a few bug fixes and improvements ) although not sure that this would have an effect on the issue you described.

[benjaminbojko]

Thanks for both of your responses.

I was testing on Bootamp with the latest Windows and AMD driver versions, using Gstreamer 1.12. I upgraded to 1.14 on my MBP and that actually broke video rendering altogether for both the GPU and CPU paths. Here's the log:

|info   | gst::video::checkBusMessagesAsync[266] Pipeline state changed from : NULL to : READY with pending : PLAYING
|info   | gst::video::checkBusMessagesSync[151] Need context : gst.gl.GLDisplay from element : convert
|info   | gst::video::checkBusMessagesSync[151] Need context : gst.gl.app_context from element : convert
|info   | gst::video::GstPlayer::setPipelineState[1053] Pipeline state about to change from : NULL to : PLAYING
|info   | gst::video::GstPlayer::setPipelineState[1056] Blocking until pipeline state changes from : NULL to : PLAYING
|error  | gst::video::GstPlayer::checkStateChange[1067] Pipeline FAILED to change state.
|error  | gst::video::checkBusMessagesAsync[195] ERROR from element qtdemux4 : Internal data stream error.
|error  | gst::video::checkBusMessagesAsync[197] Debugging info : qtdemux.c(6073): gst_qtdemux_loop (): /GstPlayBin:playbinsink/GstURIDecodeBin:uridecodebin4/GstDecodeBin:decodebin4/GstQTDemux:qtdemux4:
streaming stopped, reason not-negotiated (-4)

Any thoughts on what might be happening?

I ran another test on a Windows 10 PC with a GTX 1060 and an old i5 using Gstreamer 1.14 and everything ran like butter (1500fps with 4K@30Hz H264 footage) with ~50% CPU and ~50% GPU usage.

Could this be the classic issue of Nvidia simply being better for hardware accelerated video? We've had plenty of AMD issues with WMF as well. Would there be a way to toggle CPU/GPU paths at runtime or make this a more exposed parameter somehow? We typically use Nvidia hardware in production exclusively for exactly that reason, but are often bound to AMD via our Macbook Pros.

[PetrosKataras]

@benjaminbojko This sounds like the installation broke somehow - I would try to do a fresh install of the complete run time packages followed by the dev packages of GStreamer together with a log out / reboot of the machine just to be sure that the env is properly set.

It could be that the GL path somehow plays better on specific hardware and driver implementations. I have only tried it with Nvidia and Intel hardware and I didn't face any similar issues there. Since the GStreamer player currently is hidden behind the qtime namespace the only way to make this a runtime option would be to expose the header files of the GstPlayer to applications and then the user could directly work with the specific interface instead of having to comply with the qtime namespace.

In any case as this gets more widely used I personally start to think that probably a CMake option GSTREAMER_USE_GL would be more appropriate instead of forcing it as the default. I could provide this with a small PR if there is a general preference towards this direction.

[benjaminbojko]

@PetrosKataras Thanks for your response. I was able to do some more testing. After cleanly wiping Gstreamer and re-installing the latest Gstreamer as well generating the Cinder GS build from scratch I got video playing again, but the image using GL was sliced in half horizontally:

Any thoughts on what might be causing this? With the CPU path everything goes back to normal, but obviously the framerate drops quite a bit.

[benjaminbojko]

It turns out that the split only happens on a 2160p file I got from here: https://svs.gsfc.nasa.gov/4655

The 1080p, 720p and 360p samples from the same page play fine at the bieggning, still experience stuttering after ~10s, that I only see on my MacBook Pro with an AMD GPU.

I also tested the 720p and 1080p H264 samples (non-stereo) I from here, which have the same stuttering after ~10s: https://download.blender.org/peach/bigbuckbunny_movies/

Have you run into any limitations with resolutions and/or codecs? I did notice that the stuttering immediately starts if I go fullscreen, but I assume that's related to the good old Windows 10 fullscreen optimization issue (going windowed with -1/-1 x/y and +2 width/height fixes this).

[PetrosKataras]

Haven't run into any specific limitations so far with neither resolutions nor codecs. That said, 1080p/2160p and H264 in .mp4 containers is the combo that I personally have to deal with most often.

At this point, I would try what I suggested @richardeakin in this comment in order to try and extract more info about what is happening here. Specifically the results when disabling framerate and Vsync altogether.

Something that might also worth checking is any system wide settings that are being applied/saved from an AMD control panel or similar ( smt like NVIDIA Control Panel )

[benjaminbojko]

Ah I remember that comment and that was one of my original suspicions as well. Going fullscreen definitely causes more immediate and pronounced stuttering and playing in windowed mode is smoother for longer periods of time, but I'll try to map out a few test cases and report back here.