search

Rafostar / clapper

Level up your video experience with a modern and user-friendly media player.
https://rafostar.github.io/clapper/
GNU General Public License v3.0
773 stars 36 forks source link

Nvidia: VP9 video freezes when seeking #304

Open ak-42 opened 2 years ago

ak-42 commented 2 years ago

Hello. I am having an issue where the video output initially plays fine, but freezes when attempting to seek. If you don't try to seek, it works fine, but if you do, you get stuck on single frame while the underlying sound plays fine.

Does not happen to all videos, but to most videos that I download via yt-dlp from YouTube.

Output of mediainfo for the video ``` [user@fedora]$ mediainfo -f "Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film [aqz-KE-bpKQ].mkv" General Count : 332 Count of stream of this kind : 1 Kind of stream : General Kind of stream : General Stream identifier : 0 Unique ID : 196920577566673718651321122498339342798 Unique ID : 196920577566673718651321122498339342798 (0x9425861136D92AA2A6804B8E7BF309CE) Count of video streams : 1 Count of audio streams : 1 Video_Format_List : VP9 Video_Format_WithHint_List : VP9 Codecs Video : VP9 Video_Language_List : English Audio_Format_List : AAC LC Audio_Format_WithHint_List : AAC LC Audio codecs : AAC LC Complete name : Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film [aqz-KE-bpKQ].mkv File name extension : Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film [aqz-KE-bpKQ].mkv File name : Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film [aqz-KE-bpKQ] File extension : mkv Format : Matroska Format : Matroska Format/Url : https://matroska.org/downloads/windows.html Format/Extensions usually used : mkv mk3d mka mks Commercial name : Matroska Format version : Version 4 File size : 1558631754 File size : 1.45 GiB File size : 1 GiB File size : 1.5 GiB File size : 1.45 GiB File size : 1.452 GiB Duration : 634624 Duration : 10 min 34 s Duration : 10 min 34 s 624 ms Duration : 10 min 34 s Duration : 00:10:34.624 Duration : 00:10:34:34 Duration : 00:10:34.624 (00:10:34:34) Overall bit rate : 19647940 Overall bit rate : 19.6 Mb/s Frame rate : 60.000 Frame rate : 60.000 FPS Frame count : 38074 IsStreamable : Yes File last modification date : UTC 2022-04-09 08:21:04 File last modification date (local) : 2022-04-09 10:21:04 Writing application : Lavf59.16.100 Writing application : Lavf59.16.100 Writing library : Lavf59.16.100 Writing library : Lavf59.16.100 ErrorDetectionType : Per level 1 Video Count : 381 Count of stream of this kind : 1 Kind of stream : Video Kind of stream : Video Stream identifier : 0 StreamOrder : 0 ID : 1 ID : 1 Unique ID : 8277725086776764659 Format : VP9 Format : VP9 Commercial name : VP9 Codec ID : V_VP9 Codec ID/Url : http://www.webmproject.org/ Duration : 634566.000000 Duration : 10 min 34 s Duration : 10 min 34 s 566 ms Duration : 10 min 34 s Duration : 00:10:34.566 Duration : 00:10:34:34 Duration : 00:10:34.566 (00:10:34:34) Width : 3840 Width : 3 840 pixels Height : 2160 Height : 2 160 pixels Pixel aspect ratio : 1.000 Display aspect ratio : 1.778 Display aspect ratio : 16:9 Frame rate mode : CFR Frame rate mode : Constant Frame rate : 60.000 Frame rate : 60.000 FPS Frame count : 38074 Color space : YUV Delay : 0 Delay : 00:00:00.000 Delay : 00:00:00:00 Delay : 00:00:00.000 (00:00:00:00) Delay, origin : Container Delay, origin : Container Language : en Language : English Language : English Language : en Language : eng Language : en Default : Yes Default : Yes Forced : No Forced : No colour_description_present : Yes colour_description_present_Source : Container Color range : Limited colour_range_Source : Container Color primaries : BT.709 colour_primaries_Source : Container Transfer characteristics : BT.709 transfer_characteristics_Source : Container Matrix coefficients : BT.709 matrix_coefficients_Source : Container Audio Count : 286 Count of stream of this kind : 1 Kind of stream : Audio Kind of stream : Audio Stream identifier : 0 StreamOrder : 1 ID : 2 ID : 2 Unique ID : 3104901973177927948 Format : AAC Format : AAC LC Format/Info : Advanced Audio Codec Low Complexity Commercial name : AAC Format_AdditionalFeatures : LC Codec ID : A_AAC-2 Duration : 634624.000000 Duration : 10 min 34 s Duration : 10 min 34 s 624 ms Duration : 10 min 34 s Duration : 00:10:34.624 Duration : 00:10:34.624 Channel(s) : 6 Channel(s) : 6 channels Channel positions : Front: L C R, Side: L R, LFE Channel positions : 3/2/0.1 Channel layout : C L R Ls Rs LFE Samples per frame : 1024 Sampling rate : 48000 Sampling rate : 48.0 kHz Samples count : 30461952 Frame rate : 46.875 Frame rate : 46.875 FPS (1024 SPF) Compression mode : Lossy Compression mode : Lossy Delay : 0 Delay : 00:00:00.000 Delay : 00:00:00.000 Delay, origin : Container Delay, origin : Container Delay relative to video : 0 Delay relative to video : 00:00:00.000 Delay relative to video : 00:00:00.000 Title : ISO Media file produced by Google Inc. Default : Yes Default : Yes Forced : No Forced : No VENDOR_ID : [0][0][0][0] ```
Reproduction instructions 1. Download yt-dlp. 2. Download a video from YouTube. I used: `yt-dlp https://www.youtube.com/watch?v=aqz-KE-bpKQ` Here's my output: ``` user@fedora ~]$ cd Downloads/ [user@fedora Downloads]$ yt-dlp https://www.youtube.com/watch?v=aqz-KE-bpKQ [youtube] aqz-KE-bpKQ: Downloading webpage [youtube] aqz-KE-bpKQ: Downloading android player API JSON [youtube] aqz-KE-bpKQ: Downloading player afeb58ff [info] aqz-KE-bpKQ: Downloading 1 format(s): 315+258 WARNING: Requested formats are incompatible for merge and will be merged into mkv [download] Destination: Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film [aqz-KE-bpKQ].f315.webm [download] 100% of 1.42GiB in 00:47 [download] Destination: Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film [aqz-KE-bpKQ].f258.m4a [download] 100% of 29.34MiB in 00:02 [Merger] Merging formats into "Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film [aqz-KE-bpKQ].mkv" Deleting original file Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film [aqz-KE-bpKQ].f315.webm (pass -k to keep) Deleting original file Big Buck Bunny 60fps 4K - Official Blender Foundation Short Film [aqz-KE-bpKQ].f258.m4a (pass -k to keep) ``` 3. Open the resulting video. You should not be able to seek without freezing.
Clapper version info ``` [user@fedora]$ flatpak info com.github.rafostar.Clapper Clapper - Simple and modern GNOME media player ID: com.github.rafostar.Clapper Ref: app/com.github.rafostar.Clapper/x86_64/stable Arch: x86_64 Branch: stable Version: 0.5.2 License: GPL-3.0-or-later Origin: flathub Collection: org.flathub.Stable Installation: system Installed: 83.6 MB Runtime: org.gnome.Platform/x86_64/42 Sdk: org.gnome.Sdk/x86_64/42 Commit: 3b0e04f2f3cd8913f912722e235d55bda489192d0230665e435f87f56cc8ce26 Parent: af3d35b800a90f71d5410d404b373f4a5c2695d0bd0cee2dc743412a28f0832d Subject: Update Clapper to 0.5.2 (dad16df1) Date: 2022-06-24 09:44:44 +0000 ```
Rafostar commented 2 years ago

@ak-42

Is this with all experimental features disabled (pipewire and playbin3)? They are known to cause this kinds of problems, especially pipewire.

ak-42 commented 2 years ago

Is this with all experimental features disabled (pipewire and playbin3)? They are known to cause this kinds of problems, especially pipewire.

Yes. I did not change any settings, but I am on Fedora 36, which does use pipewire (by default).

[user@fedora ~]$ ps -e | grep pipewire
  10064 ?        00:00:05 pipewire
  10144 ?        00:00:11 pipewire-pulse
[user@fedora ~]$ pactl info | grep "Server Name"
Server Name: PulseAudio (on PipeWire 0.3.56)
Rafostar commented 2 years ago

Fedora 36, which does use pipewire (by default).

That's should be fine. Just wanted to confirm state of our experimental options.

Please share a little more info about your setup: What GPU? Is this Wayland (Fedora default) or Xorg? Also what decoder is being used during playback (name can be seen in video track selection button, next to progress bar)?

ak-42 commented 2 years ago

Decoder is nvvp9dec. Session is X11. EDIT: I also tried Wayland, and it's the same. Nvidia driver is 515.57. I have PRIME graphics, but I am using the Nvidia GPU completely (not in hybrid mode).

neofetch ``` [user@fedora ~]$ neofetch .',;::::;,'. user@fedora .';:cccccccccccc:;,. ----------- .;cccccccccccccccccccccc;. OS: Fedora Linux 36 (Workstation Edition) x86_64 .:cccccccccccccccccccccccccc:. Host: 81LK IdeaPad L340-15IRH Gaming .;ccccccccccccc;.:dddl:.;ccccccc;. Kernel: 5.18.13-200.fc36.x86_64 .:ccccccccccccc;OWMKOOXMWd;ccccccc:. Uptime: 3 hours, 48 mins .:ccccccccccccc;KMMc;cc;xMMc:ccccccc:. Packages: 2533 (rpm), 64 (flatpak) ,cccccccccccccc;MMM.;cc;;WW::cccccccc, Shell: bash 5.1.16 :cccccccccccccc;MMM.;cccccccccccccccc: Resolution: 1920x1080, 1920x1080 :ccccccc;oxOOOo;MMM0OOk.;cccccccccccc: DE: GNOME 42.3.1 cccccc:0MMKxdd:;MMMkddc.;cccccccccccc; WM: Mutter ccccc:XM0';cccc;MMM.;cccccccccccccccc' WM Theme: Adwaita ccccc;MMo;ccccc;MMW.;ccccccccccccccc; Theme: Adwaita [GTK2/3] ccccc;0MNc.ccc.xMMd:ccccccccccccccc; Icons: Tela-circle-dark [GTK2/3] cccccc;dNMWXXXWM0::cccccccccccccc:, Terminal: gnome-terminal cccccccc;.:odl:.;cccccccccccccc:,. CPU: Intel i5-9300H (8) @ 2.400GHz :cccccccccccccccccccccccccccc:'. GPU: NVIDIA GeForce GTX 1050 3 GB Max-Q .:cccccccccccccccccccccc:;,.. GPU: Intel CoffeeLake-H GT2 [UHD Graphics 630] '::cccccccccccccc::;,. Memory: 3785MiB / 7725MiB ```
Rafostar commented 2 years ago

Decoder is nvvp9dec.

Please try with:

GST_PLUGIN_FEATURE_RANK=vp9dec:300 flatpak run com.github.rafostar.Clapper

Decoder in the popover should change to vp9dec. This should play with software decoding. I want to check if this is Nvidia specific problem.

ak-42 commented 2 years ago 
GST_PLUGIN_FEATURE_RANK=vp9dec:300 flatpak run com.github.rafostar.Clapper

That works, seeking is now fine. At 4k 60fps, it noticeably struggles with the UI being slow, presumably due to software decoding, but it does work. At 1080p on my device, there's no noticeable slowdown, it all works fine.

Rafostar commented 2 years ago

but it does work

OK. So we know its nvcodec issue.

@ak-42 Could you also check if stateless decoder also has this issue?

GST_PLUGIN_FEATURE_RANK=nvvp9sldec:300 flatpak run com.github.rafostar.Clapper

If this decoder was available, it should say nvvp9sldec as decoder name in that popover (next to progress bar). Please check if nvvp9sldec (with sl in middle) is being used and if it also has this problem.

ak-42 commented 2 years ago 
GST_PLUGIN_FEATURE_RANK=nvvp9sldec:300 flatpak run com.github.rafostar.Clapper

When I run this, the decoder is shown as nvvp9sldec and the seeking works without issues.

Rafostar commented 2 years ago

When I run this, the decoder is shown as nvvp9sldec and the seeking works without issues.

In this case, you should open Clapper preferences window -> Tweaks -> Plugin ranking -> nvcodec, enable rank override and set all sl elements to a higher rank then their non-sl counterparts (e.g. 300). Restart player for the changes to be applied. This should solve your problem by using sl variants by default.

As for non-sl, those are older GStreamer implementations that AFAIK are scheduled to be deprecated in favor of sl versions in future. If you still wish for the old ones to be fixed, you should report that bug on GStreamer gitlab (as nvcodec issue, with mention that both software and sl decoders do not have this issue). That's where nvcodec plugin is maintained, thus cannot be fixed here. Although, one thing to consider for me is if I should make them default ones a little early on before GStreamer does :thinking:

most videos that I download via yt-dlp from YouTube

This is slightly unrelated to this issue, but in case of Clapper Flatpak (that you use), you can just open YT video URI or simply drag and drop video image from browser to play it. We do not have options to set codec, max res yet through (1080p60 max at the moment).

ak-42 commented 2 years ago

Thanks for the explanation. I actually checked the same video with GNOME Video (Totem) and it has a very similar seeking issue, which I suspect might also be caused by the same problem. I'll make the necessary config changes, and I look forward to a future where sl decoders are the default.

As for watching YouTube, I'd love to do that, but I am too used to Sponsoblock. yt-dlp has it implemented as well. It might actually be a cool optional addition for you to consider adding to Clapper. There's already an mpv implementation that you could check out as a reference, should you be interested.

Rafostar commented 2 years ago

(Totem) and it has a very similar seeking issue

Like I said, its GStreamer nvcodec issue, so every app that uses it will have it. I do not know if anyone reported it to GStreamer gitlab already through.

Ideally change to sl by default should happen in GStreamer, but if we want to make sl default here earlier then GStreamer does, it would be nice if few other Nvidia users could test both non-sl and sl decoders and confirm that sl ones indeed already work better.

Nonetheless, manually editing elements ranks in Clapper preferences like I described earlier should workaround this problem.

axel358 commented 2 years ago

This isn't an Nvidia exclusive bug though, I'm an AMD user and video playback freezes while seeking as well, both will hwaccel and without it. I was able to reproduce this on totem as well so I'm pretty sure this gotta be a gstreamer bug. Screenshot from 2022-08-02 10-41-55 Screenshot from 2022-08-02 10-44-56

Rafostar commented 2 years ago

@axel358

This isn't an Nvidia exclusive bug though, I'm an AMD user

No, the bug reported in this issue was Nvidia exclusive. Where old implementation that relies more on the Nvidia binary driver does not work, while newer one does. If you have problem with/without vaapi, its different one. If you are trying to play web/youtube video (what I assume from your screenshots), you should use Flatpak version of Clapper instead (only this package can currently do this right). Otherwise for local videos you can consider switching to vah264dec instead of vaapidecodebin (unless you are on Xorg), for both better performance and to avoid known vaapidecodebin issues.

axel358 commented 2 years ago

But without using vaapi i also get the freezing as shown on the other screenshot. It was a local video and yes I'm on Xorg

Rafostar commented 2 years ago

But without using vaapi i also get the freezing as shown on the other screenshot. It was a local video and yes I'm on Xorg

If its local video and happens with and without vaapi and also (as you mentioned earlier) with Totem, then you should probably report this on GStreamer gitlab. Nonetheless this bug here was Nvidia stateful implementation specific and either changing to Nvidia statless or software decoding fixes this issue, thus your one is a different GStreamer problem.

raffaem commented 1 year ago

(Totem) and it has a very similar seeking issue

Like I said, its GStreamer nvcodec issue, so every app that uses it will have it. I do not know if anyone reported it to GStreamer gitlab already through.

Ideally change to sl by default should happen in GStreamer, but if we want to make sl default here earlier then GStreamer does, it would be nice if few other Nvidia users could test both non-sl and sl decoders and confirm that sl ones indeed already work better.

Nonetheless, manually editing elements ranks in Clapper preferences like I described earlier should workaround this problem.

So, I have NVIDIA proprietary driver, clapper installed from flathub, Fedora Workstation 36.

The video is the following:

mediainfo test.webm 
General
Complete name                            : test.webm
Format                                   : WebM
Format version                           : Version 4
File size                                : 160 MiB
Duration                                 : 15 min 23 s
Overall bit rate                         : 1 453 kb/s
Writing application                      : Lavf58.76.100
Writing library                          : Lavf58.76.100

Video
ID                                       : 1
Format                                   : VP9
Codec ID                                 : V_VP9
Duration                                 : 15 min 23 s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Color space                              : YUV
Language                                 : English
Default                                  : Yes
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : Opus
Codec ID                                 : A_OPUS
Duration                                 : 15 min 23 s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Bit depth                                : 32 bits
Compression mode                         : Lossy
Language                                 : English
Default                                  : Yes
Forced                                   : No

1.

flatpak run com.github.rafostar.Clapper test.webm

Decoder: nvvp9dec.

Starts fine. Seeking has video stuck on the image I land into, while audio correctly moves forward.

Every time I seek I get the followin error in the prompt:

(com.github.rafostar.Clapper:2): GStreamer-CRITICAL **: 17:58:52.344: gst_segment_to_stream_time: assertion 'segment->format == format' failed

2.

GST_PLUGIN_FEATURE_RANK=vp9dec:300 flatpak run com.github.rafostar.Clapper

Decoder: vp9dec Seeking works fine. But the UI gets stuck when I stop it.

3.

GST_PLUGIN_FEATURE_RANK=nvvp9sldec:300 flatpak run com.github.rafostar.Clapper test.webm

Decoder: nvvp9sldec.

Seeking works fine.

All is fine.

The ranking is reversed? In a ranking the first is better than the second, no? Here it looks like the codec with the higher value of the number is chosen over the codec with the lower value of the number.

Changing the ranking to have nvvp9sldec as default works.

In all three cases, when I click on the camera to open the pop-up with the codec, the Alt+Tab GNOME shortcut to switch windows doesn't work anymore.

Rafostar commented 1 year ago

Here it looks like the codec with the higher value of the number is chosen over the codec with the lower value of the number.

Yes, element that plays the same format with higher value is chosen, how higher does not matter. This is how GStreamer is designed.

The ranking is reversed?

We try to use (in most cases at least) values as they are decided by GStreamer devs. The sl ones are considered new, so they did not set them higher then old "stable" ones yet. You can change these permanently to your own liking in Clapper preferences window. Otherwise, problems with only particular elements should be reported on GStreamer gitlab, since they are developed there.

In all three cases, when I click on the camera to open the pop-up with the codec, the Alt+Tab GNOME shortcut to switch windows doesn't work anymore.

This is a known GTK4 issue. Not related to Clapper. See: https://gitlab.gnome.org/GNOME/gtk/-/issues/4701.

raffaem commented 1 year ago

We try to use (in most cases at least) values as they are decided by GStreamer devs. The sl ones are considered new, so they did not set them higher then old "stable" ones yet. You can change these permanently to your own liking in Clapper preferences window. Otherwise, problems with only particular elements should be reported on GStreamer gitlab, since they are developed there.

I understood that.

What I mean is: if you tell me that something is a "ranking", and you use the word "ranking", what I understand is that the one ranked first is better than the one ranked second, the second is better than the third, and so on.

So I would have expected the decoder ranked first (i.e. number 1) to be preferred over the decoder ranked second (e.g. number 2).

If GStreamer do the opposite, fine. Maybe I missed that in the doc.

TheOPtimal commented 1 year ago

I have an Intel CPU, having the same issue on all codecs.

Rafostar commented 1 year ago

I have an Intel CPU, having the same issue on all codecs.

This one is for Nvidia only. If it happens with different decoders, then the cause is likely different and we would need a separate issue. Upstream already has this one: https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/1477 Maybe that's what happens to you as well? If it happens only sometimes for you, make sure you have pipewire disabled in Clapper preferences.