Use separate ffmpeg builds for non-UWP targets and properly separate builds of FFmpegInteropX.dll

[softworkz]

Related PRs

The following partial PRs are included in this PR:

• [ ] #380
• [ ] #381
• [ ] #382
• [ ] #383

Introduction

These are some of the artifacts from my ongoing struggle in hunting down a crash when consuming FFmpegInteropX in a MAUI app (WinUI3) which only manifests in combination with a WebView2, and only when the WebView2 is coming from a separate (project) assembly. Of course I had initially focused on the other parts, but then I came to a theory and I wanted to test it.

I'm adding separate messages below regarding the individual parts.

Just a note about the small commit "Add SkipBuild switch". It's simply for being able to execute/test nuget package creation without running through the whole build each time.

tbc

[softworkz]

Required Dependencies for the WinUI Package

This had driven me nuts for some time: Even though I had read the (sub-optimal) documentation on the GitHub site for CS/WinRT and the walk-through in learn.ms.com, it hasn't been clear to me that a reference to CsWinRt is required at the consuming side, even in case when a projection library is provided already. It is stated that it can(!!) be used for creating a projection library by component providers or(!!) for consuming WinRT components. This sounded like being for components which do not provide a projection lib, in which case the package would create one for you when referencing CsWinRt. But that it's even required in order to consume a package which already has a projection lib is really odd. Why can't they generate something that can be consumed without further hocus pocus...? I don't know.

To save others from this painful experience, this PR is adding the required dependencies (WinAppSDK plus WinSDK BuildTools plus CsWinRt) to the nuget package, so that it will work right out of the box.

[softworkz]

Reducing Platform Target Versions

Throughout all parts (nuget packages, builds and projects) I have reverted the version from 22000.0 back to 19041.0. There are reasons for sticking to lower versions and when nuget packages have a higher number, it is forcing you to increase the number at the consuming side in order to be able to use the package.

There's no problem in the other direction, so component packages should use rather low versions as far as possible. I haven't seen any forcing reason to got up to 22000.0, but maybe I missed it..

[softworkz]

Runtime Dependencies

Now back to my crash investigations. I had seen that the generated binaries (both FFmpegInteropX and the ffmpeg dlls) are linking to MSVC runtime versions which are specific to UWP:

I've also seen that other libs in the process are using the non-UWP versions of those libs, and that made me think that it could cause naming collisions in the process (even though on Windows it's not like on Linux where a function with a certain name can exist only once within a process. But there's a lot of things going on with all the manifest stuff and redirection mechanisms nowadays, like those:

Then I also investigated the Win2d packages (it's not easy to even find a package with native WinRT/UWP components). There's a UWP and a regular version of the Win2d package, and I compared the dependencies of the canvas dll in both cases:

• The UWP version links to the app-local runtime libs
• The regular version doesn't

Considering the above, it seemed worth to give it a try, and the changes are included in this PR:

Building FFmpeg as non-UWP versions

It wasn't as easy as I thought. I'll spare the details. Just the following note:

I had vcpkg installed and globally enabled - and I also had vcpkg packages for lzma and others, which screwed the builds - interestingly, it doesn't get active in case of the UWP builds, which made this even more annoying to find out. I've been really shocked to see that vcpkg affects ALL builds you are doing with msvc, I wasn't aware of that.

In case you are using it, I can only recommend to:

vcpkg integrate remove

and use project-based activation only.

Anyway - in the end, I was able to get all the ffmpeg dlls without any UWP dependencies.

[softworkz]

VCXPROJ - Separating UWP and WinUI Configs

The final step was to use the regular ffmpeg dlls for the WinUI builds and the UWP versions for the UWP builds when compiling the FFmpegInteropX.dll.

The default template in VS for creating a WinRT components (the say for both, WinUI3 and WinUI2/UWP), doesn't differentiate for the build outputs between the two targets and they are unconditionally including the WindowsAppSDK and SDK build tools. What is not clear though, it how this template is intended to be used. I rather suspect that you are expected to change a few things when you want to target UWP. Also, looking at Win2D kind-of proves for me that different outputs are needed. They even have two separate nuget packages.

Finally, there were also some crasshs reported (here: https://github.com/ffmpeginteropx/FFmpegInteropX/issues/378) and I'm suspecting that this might be due to the unconditional adding of WinUI settings to the build.

I have adjust the vcxproj in a way that everything regarding UWP remains same like it was before. Now, the changes for WinUI are only active for the WinUI and Desktop builds. And finally, the latter two are using the non-UWP package of ffmpeg.

Maybe not every single change in the projects is necessary, but all are taken from some MS code or documentation.

[brabebhin]

Wow, ok, this is a lot to process. Let's start from the beginning. Can you provide a repo of that MAUI app that crashes?

I see no reason why you'd need a desktop specific FFMPEG build. I am able to use the UWP ffmpeg build in winUI just fine (except the subtitle flicker but that's ffmpgeinteropx problem, not ffmpeg).

TBH, I'd be inclined to believe this is a MAUI problem, not our problem. MAUI has been a pretty consistent bug hive with no improvement in sight.

The win2D reference will be removed in the future, as the IBasicVideoEffect it is used in doesn't work with C++/WinRT or CsWinRT or whatever. Win2D has different packages because the winUI version is essentially a rewrite of the UWP version.

PS: there's currently a bug in the nuspec file on master which causes the interop library to not be included in the nuget. I am not sure when our nuget packages stopped working on x64 but it should be fixed in the pre50 nuget I released a few days ago.

[softworkz]

Wow, ok, this is a lot to process. Let's start from the beginning. Can you provide a repo of that MAUI app that crashes?

It's complicated, because a lot of components come into play here, and when removing the custom webview2 or FFmpegInteropX, it's working fine. It's also complicated, because this goes over many C# projects, which are dependent of each other and there's a plugin mechanism as the player is essentially a plugin to the client apps. Of course I have ruled out the most obvious things already, like by copying the code directly into the app projects or trying with the original FFmpeInteropX.FFmpegUWP, plus FFmpegInteropX compiled from the original winui-build branch, copying newer versions of the webview dlls into the appsdk folders, tying different appsdk and WinSDK versions, different .net versions, and all the likes. Means I'm already in the area of freaky things.

Essentially, it's like this: As soon as some class which is using FFmpegInteropX gets loaded, the Windows.Media playback gets killed. It is not even needed to execute any FFmpegInteropX method, it happens when it gets loaded.

For example, when I set a video url as source of the media element (in the main app project), it starts playing on startup as expected. But when I put some code like this into the main app:

            if (DateTime.Now.Ticks == 1)
            {
                var x = FFmpegMediaSource.CreateFromUriAsync("asaddf");
            }

The video doesn't start playing - even though the code above is never executed.

When such code is in a class in a different project, the video starts playing, but as soon as I create an instance of that class, the video stops playing and the properties of the MediaPlayer object (when inspecting in the debugger) are turning all red and inaccessible when stepping through the instance creation.

Once this has happened (in whichever way), the WevView2 is crashing sooner or later with a memory access violation exception.

When doing things normal (in the intended way, which also works in case of UWP), which involves creating a new MediaPlayer instance and setting it to the media element, playback is even working - without seeing anything, but the audio is working and the MediaPlayer instance is properly firing events etc. But it can't work together with the media element. It's like two "worlds" where one is killing the other as soon as loaded and they are unable to cooperate.

These are the things which made me look at the native side: Which dlls are loaded and are they compatible or maybe contradicting? So I came to the UWP dlls. I have seen that avcodec and FFmpegInteropX.dll are linking to VCRUNTIME140_APP.dll and that WebView2 components are using VCRUNTIME140_WIN.dll in c:\windows\system32. From the names of both, I'd assume they are doing the same thing and having both in a single process might(!!!) not be a healthy condition.

[softworkz]

PS: there's currently a bug in the nuspec file on master which causes the interop library to not be included in the nuget. I am not sure when our nuget packages stopped working on x64 but it should be fixed in the pre50 nuget I released a few days ago.

Please note that this PR is made against the winui-build branch, not the main branch. I never had an issue with the packages from the main branch. What I said above was specifically about the new WinUI package.

btw, the main branch is in pretty good shape imo, kudos!

[brabebhin]

So all those changes fix that issue?

Have you tried referencing the ffmpeginteropx project directly in winUI? There are some proj changes you need to do to reference it without the interop assembly.

[softworkz]

So all those changes fix that issue?

Nope. I wish they would. 😢

But I still think these changes are right. Even when my suspicion about invalid dll mixing would turn out to be wrong - these are still UWP libs and UWP is officially abandoned, which means that sooner or later, a clean UWP-free build will be desirable. That's why I took the time to pack them together while I'm at it, because only few hours later I might have been too far away and lost it.

Have you tried referencing the ffmpeginteropx project directly in winUI? There are some proj changes you need to do to reference it without the interop assembly.

Well, but I'd still need CsWinRT which generates interop files into the project, right?

The one thing that I'm sure about is that whatever happens is at the native side, so I don't believe that it matters whether the interop code is directly in my project or in a (also managed) interop assembly.

[softworkz]

project directly in winUI?

"WinUI directly" is a good point. That's what I'm doing right now: Recreating it all with a pure WinUI project to get MAUI out of the game 😄

[brabebhin]

Lazt time i checked MAUI, winUI had a different project than the rest of the platforms.

I expect you used the nuget package to reference ffmpeginteropx. What i proposed is to import the ffmpeginteropx project in your sln and reference it where needed. You can check our winui sample to see how it's done.

Also how exactly do those plugins get loaded? Some sort of runtime based detection with dlls?

This is why a repro sample would have been great.

Personality i think maui is not suitable for client facing applications. It works good for the average internal HR app, but not client facing. That thing is so bugged it's cheaper to just build native apps for all target platforms. There's a reason not even Microsoft is using MAUI for it's own apps.

I don't think the UWP dlls will be abandoned. Packaged WinUI is essentially UWP but with a lax sandbox. The UWP app activation model won't go anywhere soon.

[softworkz]

Lazt time i checked MAUI, winUI had a different project than the rest of the platforms.

It's all in one now. Win, Android, iOS, Mac and Tizen.

Personality i think maui is not suitable for client facing applications. It works good for the average internal HR app, but not client facing. That thing is so bugged it's cheaper to just build native apps for all target platforms. There's a reason not even Microsoft is using MAUI for it's own apps.

Agreed. It's in no way ready for creating apps with UI. But our case is special because the app is all in htmljs, so what's needed for an app is: a webview (with transparency), a player and native backend code (non-ui). We already have a dozen of apps, but the backend code is different everywhere, and unifying the latter is the opportunity which MAUI is opening up,, even though it's rather .net's multitargeting capabilities which make this possible. The whole project goes beyond maui anyway, the abstractions are our own, so that other frameworks (uwp, winui, electron.net...) are handled in the same way.

This is why a repro sample would have been great.

Working on it.. 😆

I expect you used the nuget package to reference ffmpeginteropx. What i proposed is to import the ffmpeginteropx project in your sln and reference it where needed. You can check our winui sample to see how it's done.

Your MediaPlayerWinUI sample is referencing the projection lib (FFmpegInteropX.Dotnet.dll) in release mode, and in debug mode it references the vcxproj and generates interop classes with the help of CsWinRT, which is exactly what I have described above. It is not possible to directly reference a winmd file or a C++ project without interop generation using CsWinRT. The only difference between the two ways (nuget vs. project reference) is where the interop classes are generated: either in the consuming project or in a separate assembly.

[brabebhin]

We're not using the dotnet interop project in the development loop, that's only used in the nuget package.

I was curious if there's a difference in how the interop assembly behaves between the nuget and the "direct" reference. For my own app I reference the ffmpeginteropx project the same way the winui sample does.

Btw, there's an outstanding issue with the IBasicVideoEffect in the cpp/winrt cs/winrt or whatever. When attempting to use that with a media player you will get a class not registered exception. I wonder if it has something to do with your problem.

[softworkz]

We're not using the dotnet interop project in the development loop, that's only used in the nuget package.

Your MediaPlayerWinUI sample is referencing the projection lib (FFmpegInteropX.Dotnet.dll) in release mode

[brabebhin]

Haha your comma tricked me. I edited your post to be clearer on what you mean.

Anyways, looking forward to that sample. This seems to be a new use case for our library.

[softworkz]

I don't think the UWP dlls will be abandoned. Packaged WinUI is essentially UWP but with a lax sandbox. The UWP app activation model won't go anywhere soon.

Those app-local dlls are abandoned already.

In Visual Studio, you can find three project templates for creating a C++/WinRT component. From old to new:

The first two are still using the app-local runtimes, but the third one is using the system-installed runtimes and no longer those app-local versions.

[brabebhin]

I'm not too concerned about that. This isn't android, thankfully. For now, the uwp build should work fine. I'd been quite impressed if it didn't. And I'm not easily impressed.

I actually did try that template with winui runtime component and it didn't work out well for me.

Unfortunately the tooling support for those cppwinrt and CsWinRT isn't very great. It's definitely a setback from cppcx.

[softworkz]

I'm not too concerned about that. This isn't android, thankfully. For now, the uwp build should work fine. I'd been quite impressed if it didn't. And I'm not easily impressed.

Oh no, I didn't want to imply that it won't work anymore. It will probably work for a very long time. That a point where absolutely none of the competitors can compete. Not Apple, not Google, not Linux: An application that works on Windows will keep working until the end of days (almost). The only break that they ever made was dropping support for 16bit apps. After 25 years...

But there might come new things which you can't do anymore with the old way. At some time...that's what I meant.

I actually did try that template with winui runtime component and it didn't work out well for me.

That template is essentially how the WinUI build works that @lukasf has done.

Unfortunately the tooling support for those cppwinrt and CsWinRT isn't very great. It's definitely a setback from cppcx.

Yea, what a horrible thing they did here. The past days I've gone through all this in a kind of crash-course and that's all so messed up and stupid. Alone the dysfunctional nuget integration with those import project statements for props and targets and the conditions and the extra checks for existence at the bottom... When you need to change a single import, you need to make make up to a dozen text changes. I've dropped that right away in this PR, because when a reference is not there then the project fails to load - which is still better than those tripled checks.

And what's even worse is that it seems that CsWinRT and CppWinRT are kind of dead. Looking at the commits, there's just some maintenance work happening but nothing of the things they promised in their docs. And the docs themselves (CsWinRT) are difficult to understand and there's no place where you can get an overview on this mess.

[softworkz]

Btw, there's an outstanding issue with the IBasicVideoEffect in the cpp/winrt cs/winrt or whatever. When attempting to use that with a media player you will get a class not registered exception. I wonder if it has something to do with your problem.

It's not related to my problem, but I see the same that it's not included in the projection lib. I have a multi-targeting C# project for UWP and WinUI, so it can be seen nicely here:

Comparing both shows that two types are missing:

I have no idea about the reason, but it has surely something to do with CsWinRT..

[brabebhin]

That template is essentially how the WinUI build works that @lukasf has done.

Strange, the build didn't work at all for me. But this has been some time in the past. I haven't really kept up to speed with those changes, so I may be wrong on some things. It is good that they actually work now haha.

Oh no, I didn't want to imply that it won't work anymore. It will probably work for a very long time. That a point where absolutely none of the competitors can compete. Not Apple, not Google, not Linux: An application that works on Windows will keep working until the end of days (almost). The only break that they ever made was dropping support for 16bit apps. After 25 years...

But there might come new things which you can't do anymore with the old way. At some time...that's what I meant.

I think we need to differentiate between WinUI/UWP XAML and winRT. WinUI/UWP are UI frameworks. winRT is an OS framework. Technically winUI/UWP could be ported to other systems. winRT not so much. winUI and UWP could go away any time soon, winRT is here to stay. It's like saying COM or win32 is done for.

ffmpeginteropx is mostly a winRT library. We technically could do away with any ties to the UI framework and be a fully winRT library. There will be some indirect ties to the UI framework in the winUI build because we need to pass the HWND of the window to check HDR support. But this is not tied to winUI framework.

Are the system installed dlls actually different than the app local dlls, other than path?

I have no idea about the reason, but it has surely something to do with CsWinRT..

We are actually intentionally removing it from the build from winUI because it is not working.

[softworkz]

We are actually intentionally removing it from the build from winUI because it is not working.

Perhaps it's due to the duplicate interface names: There's on IBasicVideoEffect interface from the SDK and a private one of the same name and the class is inheriting from both.

Or maybe you shouldn't expose the implementation object from the library. It would be possible to separate between a management/config class which is exposed and provides the methods to configure and apply the effect, while the IBasicVideoEffect implementation would remain private all the time.

[brabebhin]

It is ok to simply remove it. That effect was something I thought would be useful to have in order to support hardware accelerated filters. However, it has several issues, it makes MediaPlaybackList bug out randomly, it doesn't update if playback is paused but filter properties change etc. I don't think anyone uses it.

There's an open bug in cswinrt about IBasicVideoEffect. People tried using it with CameraCapture and their own custom implementation (nothing to do with ffmpeginteropx) and still doesn't work.

[lukasf]

Huh this is a lot. Just so I understand correctly: All these changes did not solve the issues you see, when using the lib in MAUI with WebView2, right? We should really think twice if we want to provide separate Desktop/UWP FFmpeg builds. I was considering it, but decided against it. And I still don't rally see a good reason now. It is not a problem to include dlls with same method names in native code. The dlls are available and not going anywhere quickly. They both do more or less the same. Just the APP ones probably have a few less of the restricted APIs.

Building ffmpeg is a pretty time consuming task now already. Doubling that plus the effort of maintaining two packages is quite a lot, when no benefits are seen. Maybe I'll give this some more thought. But right now I am not sure if it is a good idea.

The BasicVideoEffect was removed from WinUI build because of a) Win2d for WinUI is still a beta version and not much progress was seen over there and b) people had trouble getting the effect loaded (same for us). It is also not nice to force the Win2d dependency to all users if only few really use the effect. My plan is to generally remove it and maybe ship it later as a separate package for UWP (any maybe later for WinUI, if we get it working someday).

If we decide merging this, the encoder stuff should be made configurable, not just removed.

[softworkz]

Sure, I think the one part which is more or less mandatory is the inclusion of CsWinRT (and sdk references) as dependency in the WinUI nuget package and the other one is to keep out the WinAppSdk and everything related out of the UWP build.

The ffmpeg split is probably not needed (still looking into all this).

[softworkz]

It is not a problem to include dlls with same method names in native code.

On Linux, this is impossible, but it works on Windows, yes - at least basically. But there are caveats to this: When the code which is using a function in one dll is not fully isolated from code which uses the same function in the other dll and these code parts are interoperating together, then it's a different story. As soon as memory allocations are getting mixed (due to the split caused by the duplicate functions), and memory which has been allocated by one runtime is about to become freed by the free() function in another runtime, then it's all over and a crash is just around the corner. And since we are dealing with COM (its successor), we just got the opposite of "isolated code" to deal with. Probably it's all good, though..

[brabebhin]

I don't see how what you're saying here could possibly happen and we would be able to do anything about it. It sounds like you found a way for Windows to exploit itself through dll injection.

free requires a pointer to free upon, and that means a shared state between components. If the shared state exists, and one of the components frees without telling the other, I don't see how different dlls could make any difference in this. You can call free from different dlls and have the same effect on the pointer, no matter which dll malloced it.

You could argue that different dlls could be loaded and a new one would override the old one and thus function pointers inside the dll could get messed up and the free function gets called by mistake, but I see 2 problems with that: you'd have to be incredibly lucky to obtain that by mistake. The OS would simply freely load some dll and override other dlls already loaded in memory, which sounds like a security problem with the OS itself.

Now, as far as I know, webview2 processes runs in a sandboxed environment. I don't see how ffmpeginteropx could possibly interact with anything inside that environment. Webview2 should be its own process with its own dlls loaded. I'd imagine the sandboxed environment also means you can't inject random dlls inside it. If it does, again, we are back to OS bugs. Which are not to be out of the question, but I think pretty much any package that's using some random dll could cause this, and webview2 would essentially be this very brittle thing that nobody can use. If anything is causing the crash, it happens in the parent process of all these things.

As soon as memory allocations are getting mixed (due to the split caused by the duplicate functions),

I suppose you mean memory allocation of static stuff inside the dll? I am pretty sure memory that you alloc with malloc/new is in a different memory area than where functions reside, and no amount of function duplication can get those pointers mingled up. And even statically declared variables inside a dll should live in an isolated memory area.

I think of something else: loading a winRT component requires a package identity and the whatever is used to dynamically load those plug-ins at runtime does not provide that package identity, and that causes the OS to panic and cut everything. At least this is the primary difference between how we are using ffmpeginteropx and how you described it is being used here.

[softworkz]

If the shared state exists, and one of the components frees without telling the other, I don't see how different dlls could make any difference in this. You can call free from different dlls and have the same effect on the pointer, no matter which dll malloced it.

Only if it's always the same malloc and free implementations. When there are two runtimes each having its own mailloc/free, then it's a different story.

Now, as far as I know, webview2 processes runs in a sandboxed environment. I don't see how ffmpeginteropx could possibly interact with anything inside that environment.

The browser runs in its own process which again has one or more rendering sub-processes (i.e. the many chriome.exe or edge you are seeing in task manager). But what's running in the process of MY app are still two primary libs (which are again bringing in quite a number of additional dlls into MY process.

There's Microsoft.Web.Webview2Core.dll which is the layer between COM (real COM, not WinRT COM) and .NET. And there's EmbeddedBrowserWebView.dll. This is the actual WebView2 control implementation, done in C++/COM. And this is already a lot of stuff, right in MY process. What's inside the WebView2 control and doing the content, that's just the Edge browser. Yes, that's isolated and a different process, but also unrelated to this.

I think of something else: loading a winRT component requires a package identity and the whatever is used to dynamically load those plug-ins at runtime

Sorry, I forgot to answer this point: They are plugins logically, but there's no dynamic plugin loading in place (not yet at least). They are just C# projects linked with project references. Otherwise, yes, it would have definitely been something to look at.

[brabebhin]

Ah ok so everything is statically compiled there. It's good we cleared this one out.

So basically you're assuming that free and malloc in uwp dll is somehow different than the free and malloc in non uwp?(the possibility of those being different is pretty slim but we're using it as example).

As I've seen so far, there's 2 types of native functions with regards to uwp friendlies: those that are by default friendly, and those that are not.

And for those that are not, Microsoft typically created function equivalents but with different names. An example of such a function is this

https://learn.microsoft.com/en-us/windows/win32/api/fileapifromapp/nf-fileapifromapp-createfile2fromappw

[softworkz]

If the shared state exists, and one of the components frees without telling the other, I don't see how different dlls could make any difference in this. You can call free from different dlls and have the same effect on the pointer, no matter which dll malloced it.

Only if it's always the same malloc and free implementations. When there are two runtimes each having its own mailloc/free, then it's a different story.

That's btw also the very reason for which ffmpeg as its own mem management functions: that there can be mixed runtimes. Hence there's the rule that may never call free() and something allocated with av_malloc and you may never call av_free that you haven't allocated with av_malloc().

[softworkz]

So basically you're assuming that free and malloc in uwp dll is somehow different than the free and malloc in non uwp?(the possibility of those being different is pretty slim but we're using it as example).

Actually I don't think that's the case here, due to the way how things get quickly screwed right in the moment when FFmpegInteropX lib is getting loaded (without doing anything else).

[brabebhin]

I tried this dummy in my own player. I added a webview 2 on top of my xaml tree, made it open youtube. Aside from some interaction issues that i'm sure had something to do with my misuse, there have been no crashes. More so, ffmpeginteropx and webview2 can output media at the same time.

As expected, webview2 is fully sandboxed, it even has its own media playback process.

the snapshot of ffmpeginteropx used here is the winUI_build branch.

[softworkz]

I think I figured it out. And I've been on the right track regarding those app-local runtime dlls. Repro is coming...

[softworkz]

As expected, webview2 is fully sandboxed, it even has its own media playback process.

Okay, seems I haven't explained this part well enough. The "sandboxed" processes you are showing are none of our business and interest. Instead, it's about the WebView2 Control implementation which is loaded and running inside our main application process. You can compare the loaded dlls from a project with WebView2 and one without.

[softworkz]

But now for the repro sample. You can find it here: https://github.com/softworkz/MauiTest

When you run it and click the button, you will see that it either crashes or breaks into an exception which is about "activation", i.e. it cannot instantiate the FFmpegInteropX.dll.

The underlying reason for this is that the app-local dlls are missing, because the project which uses FFmpegInteropX is a regular class library, and so it doesn't have any logic to deal with that situation.

The big mistake I made then, was to include the VCRTForwarders package in the main application:

      <PackageReference Include="Microsoft.VCRTForwarders.140" Version="1.0.7" />

You can replicate this by changing the condition from false to true (in MauiTest.csproj).

And then, you are already in the situation that I described:

• When pressing the button, the video starts and you hear the sound, but you don't see any pictures

I'll wait with further discussion until you have caught up...

[softworkz]

So all those changes fix that issue?

Nope. I wish they would. 😢

Just so I understand correctly: All these changes did not solve the issues you see, when using the lib in MAUI with WebView2, right?

I'm glad to say now: Yes, this PR fixes the situation I encountered.

It just didn't manifest because I missed to remove the VCRTForwarders package reference 😅

[brabebhin]

I couldn't run your sample, something to do with local nuget packages. The reference chain is as this

MAUI <---- player project (this appears to be a pure .net class library) <--(nuget)--ffmpeginteropx.

I think the problem is less to do with ffmpeg build and more on where the cswinrt tool (or whatever is used to do interop) expects the dlls to be. I am afraid that just like with direct project reference example in our own samples, simply having them next to the exe or wherever MAUI puts them in is not correct. We have to manually recreate the expected output, whatever that may be.

Probably the easiest way is to see how other runtime components distributed as nuget packages do, but I am afraid this is the only package that I am aware of xD

It may also be that MAUI needs to be treated differently than other platforms. I'm going to try this in pure winUI and see how it goes.

[softworkz]

I couldn't run your sample, something to do with local nuget packages. The reference chain is as this

MAUI <---- player project <--(nuget)--ffmpeginteropx.

Sorry, the gitignore had excluded the package. Please pull the repo again, it should work now.

[softworkz]

For explanation: The NuGet.config file in the root folder specifies the sources for nuget packages. It includes nugetstore as first entry - which is the folder with the package. You can easily put your own generated package into this folder and then use the nuget UI in VS to change it to use your package.

But the included package is created from the winui-build branch without modifications (except the version number).

[softworkz]

this is the only package that I am aware of xD

Win2D - both the uwp and the winui version. Others can be found by skimming through issues in the CsWinRT repo.

It may also be that MAUI needs to be treated differently than other platforms. I'm going to try this in pure winUI and see how it goes.

Essentially, MAUI creates a WinUI3 app for Windows.

[brabebhin]

The chain appears to work correctly with a pure winUI project. The same chain does not appear to work correctly with MAUI projects. (this is not using your sample, but rather a new chain created from scratch)

var x = FFmpegInteropX.CharacterEncoding.GetSystemDefault();

This line of code will execute correctly in winUI (this is where it dies in the MAUI equivalent)

The winUI project template is modified as following: Removed all default nuget packages, added

    <PackageReference Include="Microsoft.WindowsAppSDK" Version="1.4.230913002" />
    <PackageReference Include="Microsoft.Windows.SDK.BuildTools" Version="10.0.25188-preview" />

Note that this is for a packaged winUI application (for unpackaged you need the VCRTFrowarders package).

For MAUI i disabled all target platforms except Windows (I see that the UI still doesn't allow you to skip macs lol)

Essentially, MAUI creates a WinUI3 app for Windows.

MAUI tries to create a WinUI3 app for Windows.

[brabebhin]

Thanks for the win2D hint. Looking at it, it seems to have a different package for MAUI. This isn't referenced directly, but it is instead bundled with other packages from other platforms.

Microsoft.Maui.Graphics.Win2D.WinUI.Desktop --version 7.0.101

This seems to have a different layout than the Microsoft.Graphics.Win2D that is used for actual winUI projects. It may be that we need to follow suit.

[softworkz]

Here's the same with WinUI3: https://github.com/softworkz/WinUITest

The behavior is identical:

Without VCRTFrowarders, it doesn't work at all With VCRTFrowarders activated it APPEARS to work. You can see video. But sooner or later it will crash. Using VCRTFrowarders in a WinAppSDK app is wrong.

Setting up the package like in this PR, is the right way.

[softworkz]

Microsoft.Maui.Graphics.Win2D.WinUI.Desktop --version 7.0.101

This seems to have a different layout than the Microsoft.Graphics.Win2D that is used for actual winUI projects. It may be that we need to follow suit.

This is something different. It's only managed code and unrelated.

[brabebhin]

You should be aware that there are slight differences between your PlayerProject (which is a pure .net class library) and a WinUI3 class library.

The winUI setup I use is exactly as the one in samples, with a desktop project and a packaging project. You seem to have some sort of common project. I am not saying your PR doesn't fix the problem. What I'm saying is that the problem is a little bit more difficult than it appears to be. What I am trying to do is to fully map the problem so that we don't break something else.

this winUi setup works for me

https://github.com/brabebhin/winUITest

[softworkz]

Correct. I'm not using a "WinUI class library", because the actual project I have is multi-targeting uap10 and net6, because the player is used in both clients.

What I'm saying is that the problem is a little bit more difficult than it appears to be.

Yes indeed 😆

What I am trying to do is to fully map the problem so that we don't break something else.

Nothing can be broken because nothing is released yet. But the current branch is breaking UWP targets. Another part which is fixed by this PR.

BTW. I've studied and researched this stuff day and night. I've looked at other packages read through CsWinRT docs and repos, but reports about MAUI regarding CsWinRT, etc. etc. etc. So, I'm pretty sure about what I've submitted here.

The one thing that is debatable in this PR is whether the separate ffmpeg build is really needed. The bottom line of this is that it is advantageous primarily for desktop use, because it gets you around the need for including VCRTForwarders - which can have really bad side effects as I've experienced. For packaged apps, the packaging will take care of this. My WebView case might be special (it's a re-developed control with hWnd rendering, because the regular WinUI3 WebView doesn't support transparency). For desktop use ("non-packaged apps") it is definitely better to not have the app-local runtime dependencies in ffmpeg and FFMpegInteropX.dll.

The other parts of this PR are needed. I'm confident that you'll come to the same conclusion - eventually. (details are always debatable, of course...)

[softworkz]

Here's a comparison of projection libraries:

From Win2D WinUI Package

FFmpetInteropX.Dotnet (with this PR)

FFmpetInteropX.Dotnet (from the 50pre package)

[brabebhin]

The one thing that is debatable in this PR is whether the separate ffmpeg build is really needed.

I don't think this is needed, the crash seems to occur way before ffmpeg is even close to being loaded in memory.

It is funny that if I take your winUI sample and replace the PlayerProject with the template created from winUI class library, it does work.

Anyways, it's up to @lukasf to decide ^^

Note that the pre50 package does not contain an official winUI build and that is expected. It is built from the master branch.

[brabebhin]

That nuget reference seems to be a reference to project reunion.

[softworkz]

I don't think this is needed, the crash seems to occur way before ffmpeg is even close to being loaded in memory.

The "crash" is simply the result of a CLASS_NOT_REGISTERED error. It's just that - an error that is not handled and looks like a crash. But it has nothing to do with the crashes I've experienced and I was talking about.

The actual crashes I had seen were caused by multiple things. For once due to the fact that I had included VCRTForwarders - as the WinUI package neither included CsWinRT, nor the WinAppSDK, nor the WinSDK BuildTools. The latter three are required at the consuming side - even CsWinRT - which is not really intuitive: you think it creates the projections and then you're done with it, but no - you need to have it at the consuming side as well. The WinAppSDK has a lot of "hidden magic" built in, so it probably does some things (which CsWinRT does) automatically, which is why it works in your template-based class library.

The VCRTForwarders are meant to forward those app-local runtimes to system-installed runtimes. After I had eliminated these dependencies from FFmpegInteropX (by compiling non-UWP/WinRT-dependent binaries), the crashes were much less frequent and even the video was visible sometimes in the MAUI app. But crashes in the WebView still happened - until I finally realized that it's still there and that I need to remove it. Since the (my) FFmpegInteropX and ffmpeg dlls were no longer depending on app-local runtimes, it means that including the VCRTForwarders also had an influence on other libraries and screwed their dependencies or loader-bindings in some way. Either webview dlls or related ones. That's why it is proven imo that this mess was related to those app-local runtimes. At the beginning I said something like it would be as if there's a collision of things from two worlds and I tend to think that those worlds were the actual WinUI/WinRT world on one side and the adapted "use WinRT APIs from outside WinRT" world.

Anyway: When all that magic stuff does it's job right (and the developer as well), then packaged apps can of course deal with the app-local stuff in the right way. But with another framework (like MAUI) on top, which has probably it's on additional magic sauce (like doing other kinds of runtime bindings), then it can go wrong easily.

I don't wish anybody else to go through the same synaptic pains which I've gone through, so I think the published nuget packages should be done in a way that they are working fine in all possible cases, rather than one or two simple demo samples.

[softworkz]

That nuget reference seems to be a reference to project reunion.

Which is the original name of the WindowsAppSDK.