RTSP stream disconnects without any error

[don-pironet-hatch]

I'm encountering an issue where my RTSP sessions disconnect from the device (some sort of IOT live camera) after a brief period, approximately 2 minutes or so. Before implementing the library I started from this FFmpeg command which works fine without any disconnects.

ffplay -fflags nobuffer -flags low_delay -tune zerolatency -probesize 32 -preset ultrafast rtsp://192.168.3.1:8554/test.mp4

FFmpegInteropX code:

            MediaSourceConfig configuration = new MediaSourceConfig()
            {
                MaxVideoThreads = 8,
                SkipErrors = uint.MaxValue,
                DefaultBufferTime = TimeSpan.Zero,
                FastSeek = true,
                VideoDecoderMode = VideoDecoderMode.ForceFFmpegSoftwareDecoder, // Can't use the GPU otherwise immediately disconnect
            };

            configuration.FFmpegOptions.Add("tune", "zerolatency");
            configuration.FFmpegOptions.Add("flags", "low_delay");
            configuration.FFmpegOptions.Add("probesize", 32);
            configuration.FFmpegOptions.Add("preset", "ultrafast");

            var decoder = await FFmpegMediaSource.CreateFromUriAsync("rtsp://192.168.3.1:8554/test.mp4", configuration);
            var mediaStreamSource = decoder.GetMediaStreamSource();
            mediaStreamSource.BufferTime = TimeSpan.FromSeconds(0);

            _mediaPlayer = new MediaPlayer();
            _mediaPlayer.Source = MediaSource.CreateFromMediaStreamSource(mediaStreamSource);
            _mediaPlayer.RealTimePlayback = true;
            _mediaPlayer.Play();

            mediaPlayerElement.SetMediaPlayer(_mediaPlayer);

There are no crashes, and the logs don't show any errors. The only observable issue is the disconnection of the RTSP stream. What could be the root of the problem, and how can I go about troubleshooting it?

[brabebhin]

The RealTimePlayback property of MediaPlayer has some funny side effects, I'd try without that.

I would also try with a MediaPlaybackItem from the FFmpegMediaSource directly (the CreateMediaPlaybackItem method) -the GetMediaStreamSource API is sort of kinda deprecated and only there because some people need it for transcoding purposes.

Do you reference our project directly or the nuget packages? The nuget packages are a little out dated while we prepare for the 2.0 release, referencing the project directly will allow you to use the latest features and bug fixes.

[don-pironet-hatch]

I switched to the CreateMediaPlaybackItem method like you suggested and I turned off the realtimeplayback. I'm now being able to use the Automatic VideoDecoder so that I use the GPU without immediately disconnecting. But still after 2 minutes there is a disconnect. I'm using the nuget packages so I will now try to remove them and do a git clone and reference the project. I will keep you up to date. Thanks for your help already.

[don-pironet-hatch]

I'm having troubles adding the project as a reference. Steps that I did:

• git clone https://github.com/ffmpeginteropx/FFmpegInteropX.git
• Right click my solution > add > existing project > Source\FFmpegInteropX.vcxproj
• In my main project > right click references > add reference > on the left side projects > tick the checkbox FFmpegInteropX

I notice that FFmpegInteropX.vcxproj has missing NuGet Packages but even after a NuGet restore the error still exist.

Severity    Code    Description Project File    Line    Suppression State
Error       Cannot resolve Assembly or Windows Metadata file 'C:\Users\laptop\Documenten\folder\FFmpegInteropX\Output\FFmpegInteropX\x86\Debug\FFmpegInteropX.winmd'    Playground  C:\Users\laptop\Documenten\folder\Playground\Playground\Playground.csproj       

Severity    Code    Description Project File    Line    Suppression State
Warning     Cannot resolve Assembly or Windows Metadata file 'C:\Users\laptop\Documenten\folder\FFmpegInteropX\Output\FFmpegInteropX\x86\Debug\FFmpegInteropX.winmd'    Playground  C:\Users\laptop\Documenten\folder\Playground\Playground\Playground.csproj       
Severity    Code    Description Project File    Line    Suppression State
Error   CS0006  Metadata file 'C:\Users\laptop\Documenten\folder\FFmpegInteropX\Output\FFmpegInteropX\x86\Debug\FFmpegInteropX.winmd' could not be found    Playground  C:\Users\laptop\Documenten\folder\Playground\Playground\CSC 1   Active

Severity    Code    Description Project File    Line    Suppression State
Error       This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them.  For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is ..\packages\Microsoft.Windows.CppWinRT.2.0.220608.4\build\native\Microsoft.Windows.CppWinRT.props.    FFmpegInteropX  C:\Users\laptop\Documenten\folder\FFmpegInteropX\Source\FFmpegInteropX.vcxproj  312 

EDIT

I removed this from the project:

  <Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
    <PropertyGroup>
      <ErrorText>This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them.  For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText>
    </PropertyGroup>
    <Error Condition="!Exists('..\packages\Microsoft.Windows.CppWinRT.2.0.220608.4\build\native\Microsoft.Windows.CppWinRT.props')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\Microsoft.Windows.CppWinRT.2.0.220608.4\build\native\Microsoft.Windows.CppWinRT.props'))" />
    <Error Condition="!Exists('..\packages\Microsoft.Windows.CppWinRT.2.0.220608.4\build\native\Microsoft.Windows.CppWinRT.targets')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\Microsoft.Windows.CppWinRT.2.0.220608.4\build\native\Microsoft.Windows.CppWinRT.targets'))" />
    <Error Condition="!Exists('..\packages\Microsoft.Windows.SDK.BuildTools.10.0.22621.756\build\Microsoft.Windows.SDK.BuildTools.props')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\Microsoft.Windows.SDK.BuildTools.10.0.22621.756\build\Microsoft.Windows.SDK.BuildTools.props'))" />
    <Error Condition="!Exists('..\packages\Microsoft.Windows.SDK.BuildTools.10.0.22621.756\build\Microsoft.Windows.SDK.BuildTools.targets')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\Microsoft.Windows.SDK.BuildTools.10.0.22621.756\build\Microsoft.Windows.SDK.BuildTools.targets'))" />
    <Error Condition="!Exists('..\packages\Win2D.uwp.1.26.0\build\native\Win2D.uwp.targets')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\Win2D.uwp.1.26.0\build\native\Win2D.uwp.targets'))" />
    <Error Condition="!Exists('..\packages\FFmpegInteropX.FFmpegUWP.5.1.100\build\native\FFmpegInteropX.FFmpegUWP.targets')" Text="$([System.String]::Format('$(ErrorText)', '..\packages\FFmpegInteropX.FFmpegUWP.5.1.100\build\native\FFmpegInteropX.FFmpegUWP.targets'))" />
    <Error Condition="!Exists('..\..\Playground\packages\Microsoft.Windows.CppWinRT.2.0.220608.4\build\native\Microsoft.Windows.CppWinRT.props')" Text="$([System.String]::Format('$(ErrorText)', '..\..\Playground\packages\Microsoft.Windows.CppWinRT.2.0.220608.4\build\native\Microsoft.Windows.CppWinRT.props'))" />
    <Error Condition="!Exists('..\..\Playground\packages\Microsoft.Windows.CppWinRT.2.0.220608.4\build\native\Microsoft.Windows.CppWinRT.targets')" Text="$([System.String]::Format('$(ErrorText)', '..\..\Playground\packages\Microsoft.Windows.CppWinRT.2.0.220608.4\build\native\Microsoft.Windows.CppWinRT.targets'))" />
  </Target>

I can build the code with following errors:

Severity  Code  Description Project File  Line  Suppression State
Error C1083 Cannot open include file: 'libavcodec/avcodec.h': No such file or directory FFmpegInteropX  C:\Users\laptop\Documenten\folder\FFmpegInteropX\Source\pch.h  57  

Severity    Code    Description Project File    Line    Suppression State
Error       PRI175: 0x80070002 - Processing Resources failed with error: The system cannot find the file specified. Playground  C:\Users\laptop\Documenten\folder\Playground\Playground\GENERATEPROJECTPRIFILE      

Severity    Code    Description Project File    Line    Suppression State
Error       PRI252: 0xdef00071 - File C:\Users\laptop\Documenten\folder\FFmpegInteropX\Output\FFmpegInteropX\x86\Debug\FFmpegInteropX.pri not found.    Playground  C:\Users\laptop\Documenten\folder\Playground\Playground\GENERATEPROJECTPRIFILE      

Do I need to custom build FFmpeg because to me it seems only needed when you want to use a custom version?

[brabebhin]

In order to directly add the project, you need to follow these steps:

1. Clone the project (you did)
2. Copy the project and its source (the Source folder in our repo) to your solution directory.
3. Add the FFmpegInteropX as reference to your application. If you use UWP, this is it.
4. If your application is using winUI 3, add the following lines to your csproj. You also need to install the CsWinRT package.

<PropertyGroup>
    <!--If CsWinRTIncludes is not specified, generate projections for all referenced winmd files-->
    <CsWinRTIncludes Condition="'$(CsWinRTIncludes)' == ''">@(CsWinRTRemovedReferences->'%(FileName)')</CsWinRTIncludes>

</PropertyGroup>

<Target Name="IncludeFFmpegInteropXInRuntimesFolder" BeforeTargets="GetCopyToOutputDirectoryItems" Condition="'$(Configuration)'=='Debug'">
    <ItemGroup>
        <AllItemsFullPathWithTargetPath Include="$(OutDir)FFmpegInteropX.dll">
            <TargetPath>runtimes\win10-$(Platform)\native\FFmpegInteropX.dll</TargetPath>
            <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
        </AllItemsFullPathWithTargetPath>
        <AllItemsFullPathWithTargetPath Include="$(OutDir)FFmpegInteropX.pdb">
            <TargetPath>runtimes\win10-$(Platform)\native\FFmpegInteropX.pdb</TargetPath>
            <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
        </AllItemsFullPathWithTargetPath>
    </ItemGroup>
</Target>

@lukasf should we release a package off the current winUI config branch?

[don-pironet-hatch]

I will try that this evening. We are using WinUI 2. That would be nice if the latest version could be released.

[brabebhin]

Winui 2 as in UWP?

[don-pironet-hatch]

Yes it's a UWP app.

[brabebhin]

Ok, then simply reference the project. You also need to install VS features to build. Check the readme

[lukasf]

The reason why playback stops is that you don't store the FFmpegMediaSource instance ("var decoder" in your code) in a field. The current official build still has the restriction that you must keep a reference to the FFmpegMediaSource as long as you play it, to prevent it from being garbage collected. If you don't, playback will just stop after a random time, often about a minute or two.

The upcoming version 2.0 won't have this restriction anymore, but it is not finished yet. So just store your decoder in a field instead of a local var, then playback should work fine.

[don-pironet-hatch]

@brabebhin I've installed everything according to the readme. Just that you know the links to download the Visual C++ Redistributable for Visual Studio 2010 are not working anymore. I've found those online and when I try to install those it mentions that I already have a newer version. Don't know if this is a problem or not. Because I would still like to reference the latest version to see the difference and see debug information.

@lukasf Thank you so much this indeed solves the problem. I mistakenly believed (a rookie mistake), that it wouldn't be subject to garbage collection since it was being utilized by MediaSource.CreateFromMediaStreamSource. Any timings when the 2.0 version is going to be released?

[lukasf]

I think you don't need the C++ redists anymore, since Microsoft now has these all-in-one redist. Also, they were only required for building ffmpeg itself, not for opening or building our lib. The only things important for the lib are the UWP workload plus the UWP c++ build tools. What happens when you directly open the FFmpegInteropX.sln file?

About the 2.0 version, there are still some things that need to be decided, especially regarding the nuget package architecture. We want to fully support Desktop and WinUI 3.0 apps, and that requires some considerations. I hope we will see it this year, but I won't make promises.

[brabebhin]

Good catch with the GC, completely forgot how far behind the live version is!