Frequent failure to play hevc video

[ZzzzzzzSkyward]

With the prerelease 1.1.0-pre42, when playing local dash hevc video, two circumstances often happens:

1. The MediaPlayer starts to play and instantly ends without actually displaying anything.

2. After a delay of 2~5s, the MediaPlayer displays video, but before that it is frozen. And I can't jump to another position otherwise 1 happens. However, with version 1.0.0, or with h.264 format this doesn't happen. And because version 1.0.1 crashes when disposing stream video so I skipped testing it.

   <UserControl
   x:Class="BiliLite.Controls.Player"
   xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
   xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
   xmlns:local="using:BiliLite.Controls"
   xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
   xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
   
   mc:Ignorable="d"
   d:DesignHeight="300"
   d:DesignWidth="400">
   
   <Grid>
       <!--播放器-->
       <MediaPlayerElement x:Name="mediaPlayerVideo" ></MediaPlayerElement>
       <!--音视频分离-->
       <MediaPlayerElement x:Name="mediaPlayerAudio"  Visibility="Collapsed"></MediaPlayerElement>
       <!--<vlc:VideoView x:Name="vlcVideoView" Initialized="vlcVideoView_Initialized"></vlc:VideoView>-->
   </Grid>
   </UserControl>

   var videoFile = await StorageFile.GetFileFromPathAsync(dashPlayUrlInfo.Video.Url);
   var _ffmpegMSSVideo = await FFmpegMediaSource.CreateFromStreamAsync(await videoFile.OpenAsync(FileAccessMode.Read), _ffmpegConfig);
   var _playerVideo = new MediaPlayer();
   _playerVideo.Source = _ffmpegMSSVideo.CreateMediaPlaybackItem();
   _playerVideo.MediaEnded += new TypedEventHandler<MediaPlayer, object>(async (e, arg) =>
   {
   await this.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
   {
   });
   });
   _playerVideo.MediaFailed += new TypedEventHandler<MediaPlayer, MediaPlayerFailedEventArgs>(async (e, arg) =>
   {
   if (_playerVideo == null || _playerVideo.Source == null) return;
   await this.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
   {
   });
   });
   mediaPlayerVideo.SetMediaPlayer(_playerVideo);

[brabebhin]

Do you keep a reference to the FfmpegMediaSource object somewhere global?

Please upload a video file to onedrive and share the link if possible.

[ZzzzzzzSkyward]

video.zip Yes it is stored in the class as ·private FFmpegMediaSource _ffmpegMSSAudio;

[ZzzzzzzSkyward]

Screen captures to demonstrate the difference: v1.1.0-pre42 with 2 different failed attempts.

https://github.com/ffmpeginteropx/FFmpegInteropX/assets/55184223/059a93cf-30f2-4310-ba8a-2381308b3b26

v1.0.0 with 3 times all successful.

https://github.com/ffmpeginteropx/FFmpegInteropX/assets/55184223/69349dc2-17af-452f-96d9-d505b8840625

[lukasf]

I do not get any playback or seek issues when playing the file using our sample apps. I guess it is a problem in your app code.

While it is hard to diagnose without having the actual sources, this line looks suspicious to me:

var _ffmpegMSSVideo = await FFmpegMediaSource.CreateFromStreamAsync(await videoFile.OpenAsync(FileAccessMode.Read), _ffmpegConfig);

You say you have a field private FFmpegMediaSource _ffmpegMSSAudio; (and probably private FFmpegMediaSource _ffmpegMSSVideo; as well). But you assign the FFmpegMediaSource to var _ffmpegMSSVideo, which means that the instance will be stored in a local variable ("var") with the same name and not in a class instance field. Make sure that you really store your instances in the class (remove "var" if this is in your actual code) and use a debugger to verify that the field is still assigned when the error occurs.

You can also try to play the file with our sample apps and check if you get playback errors as well.

[ZzzzzzzSkyward]

Sorry for the mistake, but I really store all of my ffmpeg media as private class members. I can't build a sample app because I don't want to download WinRT package. So I write a small app myself to reproduce the issue. More information if this is a platform issue: I'm using windows10 22H2 19045.2846, AMD CPU 5700g, no more video card. FFMpeg is set to passthrough. Test shows that software decode is OK. 1.zip

<Page x:Class="MediaPlayerWinUI.MainPage" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:local="using:MediaPlayerWinUI">
    <Grid Background="Red">
        <Button x:Name="button" Content="Native" FontSize="20" HorizontalAlignment="Left" Margin="0" VerticalAlignment="Top" Click="Open_Click" Width="100" Height="100" />
        <Button x:Name="button2" Content="FFMpeg" FontSize="20" HorizontalAlignment="Right" Margin="0 0 0 100" VerticalAlignment="Top" Click="Open2_Click" Width="100" Height="100" />
        <MediaPlayerElement x:Name="Player" AreTransportControlsEnabled="True" AutoPlay="True" Margin="67,94,121,116" />
    </Grid>
</Page>

using System;
using Windows.Media.Core;
using Windows.Media.Playback;
using Windows.Storage.Pickers;
using Windows.Storage;
using FFmpegInteropX;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml;

namespace MediaPlayerWinUI
{
    public sealed partial class MainPage : Page
    {
        private FFmpegMediaSource FFmpegMSS;
        private FFmpegMediaSource actualFFmpegMSS;
        private StorageFile currentFile;
        private MediaPlaybackItem playbackItem;
        private MediaPlayer mediaPlayer;
        public MainPage()
        {
            this.InitializeComponent();
        }

        private async void Open_Click(object sender, RoutedEventArgs e)
        {
            var openPicker = new FileOpenPicker();
            openPicker.ViewMode = PickerViewMode.Thumbnail;
            openPicker.SuggestedStartLocation = PickerLocationId.VideosLibrary;
            openPicker.FileTypeFilter.Add(".mp4");
            openPicker.FileTypeFilter.Add(".mkv");
            openPicker.FileTypeFilter.Add(".m4s");

            var file = await openPicker.PickSingleFileAsync();
            if (file != null)
            {
                var stream = await file.OpenAsync(FileAccessMode.Read);
                Player.MediaPlayer.Source = MediaSource.CreateFromStream(stream, file.ContentType);
            }
        }

        private async void Open2_Click(object sender, RoutedEventArgs e)
        {
            var openPicker = new FileOpenPicker();
            openPicker.ViewMode = PickerViewMode.Thumbnail;
            openPicker.SuggestedStartLocation = PickerLocationId.VideosLibrary;
            openPicker.FileTypeFilter.Add(".mp4");
            openPicker.FileTypeFilter.Add(".mkv");
            openPicker.FileTypeFilter.Add(".m4s");
            var file = await openPicker.PickSingleFileAsync();
            if (file != null)
            {
                var stream = await file.OpenAsync(FileAccessMode.Read);
                if (this.Player.MediaPlayer != null)
                {
                    this.Player.SetMediaPlayer(null);
                }
                // 加载视频流到播放器
                var _ffmpegConfig = new MediaSourceConfig();
                _ffmpegConfig.VideoDecoderMode = VideoDecoderMode.ForceSystemDecoder;
                if (FFmpegMSS != null)
                {
                    FFmpegMSS.Dispose();
                    FFmpegMSS = null;
                }
                FFmpegMSS = await FFmpegMediaSource.CreateFromStreamAsync(stream, _ffmpegConfig);

                var x = FFmpegMSS.CreateMediaPlaybackItem();
                if (mediaPlayer != null)
                {
                    mediaPlayer.Dispose();
                    mediaPlayer = null;
                }
                mediaPlayer = new MediaPlayer();
                mediaPlayer.Source = x;
                this.Player.SetMediaPlayer(mediaPlayer);
                this.Player.MediaPlayer.Play();
            }
        }
    }
}

[ZzzzzzzSkyward]

I also tested v1.0.1 and v1.0.2-pre, the former runs well, but the latter fails. Hope this may help you find what is wrong. I guess during the two releases ffmpeg has been updated to not support some type of video or GPU?

[brabebhin]

I also cannot reproduce the issue, in both samples and my own app. Eyeballing your code, you may be suffering from OS execution segmentation. You should be calling SetMediaPlayer on MediaPlayerElement before setting the source of the MediaPlayer object. The real life time span between setting source and SetMediaPlayer is undetermined.

[ZzzzzzzSkyward]

If it works on your machine then I guess it is something wrong with my laptop and desktop, which both uses AMD GPU. I don't think it is because of my code, changing the order of sentences doesn't help. Luckily I can still have software decoding as an option.

[brabebhin]

It might be driver related, true. Unfortunately we can't really debug the system decoders. Our preferred solution is to use directx decoder or software decoder. The system stuff is mostly for backwards compatibility with the microsoft fork and some really isolated cases like xbox.

[lukasf]

Is there a specific reason why you want to force use of OS codecs? What happens on your system, when you use "Automatic" instead? Automatic should use the GPU as well (if the codec is supported), through D3D11 interfaces instead of OS codecs. This is the preferred configuration and usually the most reliable.

After playback starts in "Automatic" mode, you can query CurrentVideoStream.DecoderMode to check if HW decoding is in use.

[ZzzzzzzSkyward]

Using VideoDecoderMode.AutomaticSystemDecoder, it shows ffmpeg software decoder. Using VideoDecoderMode.Automatic, it shows d3d11 decoder. There is no reason why I must force system codecs, just thinking that it used to work, but not working with more up-to-date release. I can always use other codecs.

[brabebhin]

When i started off the directx implementation, i envisioned the system decoders would die out. We keep them for backwards compatibility and there's some weird scenarios in which directx doesn't work but system does, like xbox. Otherwise we fully commit to directx and we can actually debug issues with that decoder, which we can't do for system decoders.

[softworkz]

The "system decoders" are the exact same (hardware) decoders that FFmpegInteropX (i.e. ffmpeg) is using: D3D11VA decoders.

One reason for crashes / non-working FFmpegInteropX decoding is fixed here: https://github.com/ffmpeginteropx/FFmpegInteropX/pull/417