Vasilich commented 5 years ago

Reduce latency / Drop frames when CPU is too slow

When showing several streams in one window (e.g monitoring system), the low latency is very important. If CPU is too slow to show e.g. 16 streams in HD simultaneously, it is better (for such kind of applications) to skip frames, but always show the latest frame possible. Yes, I have read https://github.com/unosquare/ffmediaelement/issues/178 and tried suggested solution with new API, but it doesn't solve this problem (as the cause is different).

Issue Categories

[ ] Bug
[ ] Feature Request
[x] Question
[ ] Not sure

Version Information

[x] NuGet Package 2.80

Steps to Reproduce

put 16 Mediaelement controls on one window, filling it as much as possible
give them all the same stream URL (e.g. https://mcdn.daserste.de/daserste/de/master_3744.m3u8?sd=10&rebase=on )
make window maximized on screen
check the difference in rendered elements for the same stream (in my case - up to 7 seconds)

Expected Results

difference in 1-2 seconds is acceptable (due to buffering)

Sample Code

XAML

<Window x:Class="WpfTest.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
        xmlns:views1="clr-namespace:Bfe.Controls.MediaPlayer.Views;assembly=Bfe.Controls.MediaPlayer"
        mc:Ignorable="d"
        Title="MainWindow" Height="450" Width="800"
        x:Name="Main">
    <Grid>
        <Grid.RowDefinitions>
            <RowDefinition Height="*" />
            <RowDefinition Height="*" />
            <RowDefinition Height="*" />
            <RowDefinition Height="*" />
            <RowDefinition Height="30" />
        </Grid.RowDefinitions>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="*" />
            <ColumnDefinition Width="*" />
            <ColumnDefinition Width="*" />
            <ColumnDefinition Width="*" />
        </Grid.ColumnDefinitions>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="0" Grid.Column="0"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="0" Grid.Column="1"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="0" Grid.Column="2"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="0" Grid.Column="3"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="1" Grid.Column="0"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="1" Grid.Column="1"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="1" Grid.Column="2"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="1" Grid.Column="3"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="2" Grid.Column="0"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="2" Grid.Column="1"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="2" Grid.Column="2"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="2" Grid.Column="3"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="3" Grid.Column="0"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="3" Grid.Column="1"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="3" Grid.Column="2"/>
        <views1:MediaPlayerView SourceUrl="{Binding Uri}" Grid.Row="3" Grid.Column="3"/>
        <StackPanel Orientation="Horizontal" Grid.Column="0" Grid.Row="4" Grid.ColumnSpan="4">
            <Button Command="{Binding ChangeUriCommand}" CommandParameter="C:\Users\Public\Videos\Sample Videos\Wildlife.wmv">Play File</Button>
            <Button Command="{Binding ChangeUriCommand}" CommandParameter="https://zdf1314-lh.akamaihd.net/i/de14_v1@392878/index_3096_av-b.m3u8?sd=10&amp;rebase=on&amp;id=">Play Stream 1</Button>
            <Button Command="{Binding ChangeUriCommand}" CommandParameter="https://mcdn.daserste.de/daserste/de/master_3744.m3u8?sd=10&amp;rebase=on">Play Stream 2</Button>
            <Button Command="{Binding ChangeUriCommand}" CommandParameter="http://swrrp-lh.akamaihd.net/i/swrrp_live@196739/index_3584_av-p.m3u8?sd=10&amp;rebase=on">Play Stream 3</Button>
        </StackPanel>
    </Grid>
</Window>

C

MainWindow.xaml.cs

private ICommand changeUriCommand;
        private string uri;

        public ICommand ChangeUriCommand => changeUriCommand ?? (changeUriCommand = new DelegateCommand<string>(ChangeUriExecute));

        private void ChangeUriExecute(string arg)
        {
            Uri = arg;
        }

        public string Uri
        {
            get => uri;
            set
            {
                if (Equals(value, uri)) return;
                uri = value;
                OnPropertyChanged();
            }
        }

        public event PropertyChangedEventHandler PropertyChanged;

        [NotifyPropertyChangedInvocator]
        protected virtual void OnPropertyChanged([CallerMemberName] string propertyName = null)
        {
            PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(propertyName));
        }

Command line ffplay

ffplay -i "https://mcdn.daserste.de/daserste/de/master_3744.m3u8?sd=10&amp;rebase=on" -fflags nobuffer -flags low_delay -an -framedrop

Is it possible to implement "-framedrop" key from ffplay, or somehow reduce latency when heavy CPU load happens?

mariodivece commented 5 years ago

The aurrent architecture does not allow this but I will consider it. You can limit the FPS using RendererOptions. If I recall correctly, the option is called VideoFrameRefreshLimit or similar. This won't prevent frames from getting processed by the FrameDecodingWorker it will just prevent frames from being copied on to the Visual at a rate beyond what you specify.

Vasilich commented 5 years ago

i have tried already to set Media.RendererOptions.VideoRefreshRateLimit = 10; in OnMediaOpening, but it looked even worse than without it. The point is, that for real-time streams in monitoring app there is a need to see what is happening now, and not what has happened 20 seconds ago. Thanks for considering to find a way to solve my problem.

mariodivece commented 5 years ago

I will consider this.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.