Create WinUI build config

[lukasf]

This PR adds a true WinUI build config:

• Use WinUI DispatcherQueue and DispatcherTimer to allow smooth subtitles even on WinUI
• Added file name APIs to FFmpegMediaSource and FrameGrabber
• Disabled BasicVideoEffect in WinUI build

Currently, this uses a dynamic detection of WinUI availability. The same dll can be used on a pure Desktop/Win32 app without WinUI, e.g. for pure transcoding scenarios, etc. Not sure if this is the best approach though.

Also, I try to get the main window from the current thread on WinUI, to decide if HDR shall be used or not. This won't work reliable if we have multiple windows on the same thread (which is supported on WinUI). If we want to allow the user to specify the target window, we'd have to add either a HWND (IntPtr) or WindowId type parameter to the config class.

Open questions:

How do we want to package this? We could put both UWP and WinUI/Desktop into a single NuGet package. Or we create separate packages for each target platform.

Also, shall we continue with dynamic detection of WinUI? Or shall we rather create separate builds for WinUI and Win32/Desktop (without WinUI)? It might be best to create separate builds right away, to avoid having to split up again at a later point.

[brabebhin]

Nice work! I won't be able to test this until later this week.

I think we should have different nugets and different builds between winUI, desktop and UWP. This is what microsoft does with winui 2/3 and win2D.

[brabebhin]

If we are targeting winUI 3 without UWP support, it might be possible to get HDR support with directX desktop only APIs. I looked into this a while back and it was not possible to get HDR with DirectX in UWP, but it should be possible in desktop/winUI 3. I will dedicate some time to this in the following days.

[lukasf]

I cannot imagine how that is supposed to work. There can be multiple Windows on the current thread. Some Windows might be on a HDR display, others on a SDR display. We need to know HDR support when creating the media source, before the source is assigned to any MediaPlayerElement.

This will only work properly if the user provides the HWND or WindowId of the target Window upfront. We still cannot support the case where the Window is moved from a SDR to HDR display or vice versa, or when changing the HDR mode, because the EncodingProperties cannot be changed after the MediaSource was created.

[lukasf]

One other thing to decide: Do we still want the hard dependency from FFmpegInteropX to our FFmpeg build? It is kind of inflexible. Imagine we add transcoding support through ffmpeg. Most people just use the lib for decoding. Only some might need encoders. Some might want LGPL, some might also want GPL encoders. If we'd remove the dependency, we could provide different FFmpeg builds and devs would reference the package that fits their needs best. It would also make it easier for those that roll their own internal builds.

[brabebhin]

https://learn.microsoft.com/en-us/windows/win32/api/dxgi1_6/nf-dxgi1_6-idxgioutput6-getdesc1

This should allow us to detect if the directx output device supports HDR.

The problem of moving from a SDR window to a HDR window is probably easier to solve by clients. We need to create a way to quickly recreate the media source somehow.

I wonder though, wouldn't setting HDR properties on a media sample also work? This seems to be a random limitation in MF. Have you tried this when implementing HDR? If not, i could take a look.

[lukasf]

Okay that might work. Actually, I just saw that I do set per-frame HDR metadata, but only if the frame contains HDR metadata as frame side data. I saw this somewhere else, but could not find a file where metadata changes pre-frame, so I could not test this. But it indicates that setting metadata on sample instead of mediatype might be supported.

It would sure be great if we'd support dynamic switching here. Because on Windows, all SDR content looks really bad as soon as you endable HDR. So I usually have it disabled and only enable on-demand (for games or movies) using keyboard shortcut. If you want to look into this, that would be cool.

[brabebhin]

Yes, I will take a look at it. However, I do not see this as a blocker for a winUI specific build. Getting the flickering fixed sooner is more important imo.

One other thing to decide: Do we still want the hard dependency from FFmpegInteropX to our FFmpeg build? It is kind of inflexible. Imagine we add transcoding support through ffmpeg. Most people just use the lib for decoding. Only some might need encoders. Some might want LGPL, some might also want GPL encoders. If we'd remove the dependency, we could provide different FFmpeg builds and devs would reference the package that fits their needs best. It would also make it easier for those that roll their own internal builds.

How would this work, exactly? Wouldn't we be breaking things for people who want to keep it simple?

[brabebhin]

Anyways, so how this is going down is that I've managed to correctly enum all the displays and adapters on my machine, however, finding the current display is no trivial task. Each display adapter provides the desktop coordinates which it displays, and based on that and our own window coordinates, we can extrapolate to which window we can output HDR.

IMO, this is overly complicated for a library such as ours to fix, because we need to know where our samples will end up. What I propose:

1. We go down the window ID route. We could make the Window ID a parameter in our CreateFrom* methods to tell people this is an important one.
2. We check HDR support on every sample to handle the situation when window moves from one display/GPU to another.

[lukasf]

I played a bit with this. DisplayInformation has ...Changed events which allow you to react on HDR color changes. This works when I switch HDR on and off. Then I tried to set the HDR metadata on per-frame level, but unfortunately it is not picked up. The metadata must be set in the EncodingProperties, upfront, nothing else works. So this means, the app devs must themselves monitor DisplayInformation and restart playback in case of HDR mode changes.

We should still add overloads with WindowId, so devs can specify the target window if they have a multi window app. For single window apps this is not needed. I can reliably get the main window using Win32 APIs.

[brabebhin]

It's disappointing that we can't set the meta per frame. HDR support seems to be all over the place with Windows 11. The system should automatically pick those up.

[lukasf]

One other thing to decide: Do we still want the hard dependency from FFmpegInteropX to our FFmpeg build?...

How would this work, exactly? Wouldn't we be breaking things for people who want to keep it simple?

I just found out that it is quite easy to use custom ffmpeg dlls even, with the official nuget package (which includes dependency to FFmpegUWP). The app project just needs to add a manual reference to FFmpegUWP and then set ExcludeAssets=all.

    <PackageReference Include="FFmpegInteropX.FFmpegUWP">
      <Version>5.*</Version>
      <ExcludeAssets>all</ExcludeAssets>
    </PackageReference>

It is a bit counter-intuitive to add a reference for something you don't want to use. But it does work. Only downside is that this will make VS download the NuGet package, but I think that's acceptable.

[brabebhin]

I might be seeing things, but subtitles still flicker on winUI even with the new DispatcherQueue plugged in?

[lukasf]

I see two issues with subtitles on WinRT:

1. The anti flicker solution does not seem to work on WinRT. I am not even sure if it has an effect at all. I played a bit with the code and found a possible solution: We have this ReferenceTrack mechanism, which was added later, and is currently only used for image based subtitles (formats where sub cues have infinite duration and are replaced by next subtitle when it comes). If I just add all the subs through the RefTrack, I don't get any flicker. I don't even need the whole dispatcher stuff then. This also seems to work on UWP, and would simplify the code considerably. I need a bit more testing, but I might go that route.

2. Image based subtitles don't work at all. I have tried a lot of stuff, but image subs just don't show up on WinUI. Only in two occasions, I suddenly hat a line of sub on screen, with correct content, position and timing. I think both times, it was after restoring a playing but minimized player window. But I could never reproduce. I am not sure if there is anything we can do about that. I tried to disable the on-demand decoding, it did not make any difference. Also when adding image-based subs as external subtitles, fully decoded upfront, nothing showed up. That's quite a bummer.

[brabebhin]

I've had a look at the microsoft fork. It shouldn't be too difficult to implement pass through for subtitles.

[lukasf]

Yeah I also think it's not such a big deal. Only difficulty is the fact that it's async. But it shouldn't be that difficult. We have task_completion_source and stuff like that available.

But the current solution allows us to dynamically add new (external) subtitle streams during playback. This is not possible with passthrough approach, because we cannot add new streams to the MSS after it is created. So I hope that ImageSubtitles will be fixed at some point in WinUI, to allow us using both approaches.

[brabebhin]

External subtitles shouldn't be affected by this. We can still use our current decoding loop which essentially processes the entire file in one go, They go in the MediaPlaybackItem, not in the MediaStreamSource.

However, the key point to using MSS for subtitles is that we get rid of the dependency to the UI platform entirely. This will allow us to use a single package for all platforms (UWP, winUI, Desktop).

I will branch off this one and do the work on subs and see how it works out. It may have absolutely zero effect on the image subs though.

[lukasf]

But if we load image subs as external subs and add them to the MediaPlaybackItem, they won't work on WinUI. I already tested that. That's why I said, using passthrough won't fix the WinUI ImageSub bug for the external subtitle scenario.

I think we should still have separate builds, at least for UWP, because on WinUI/Desktop we want to support file name based APIs, which are not possible on UWP.

[brabebhin]

Yes, image subtitles are a different problem altogether now. I have doubts they will work even with winRT APIs. But for any other type of subtitle, it should be possible to add the external files without any issues.

Technically the file name API should also work with UWP, provided the paths are contained within the ApplicationData folder. I can see some use cases for that.

[brabebhin]

In the end, it is probably better to keep our current approach with subtitles if we can get anti flicker to work again. We would lose quite a bunch of features if we go the passthrough way:

1. Auto correcting ASCII to UTF 8 (this one is huge actually)
2. Custom styles
3. We would still need to maintain the same code for external subs parsing through ffmpeg.

[lukasf]

I have pushed a branch winui-subtitles. The whole dispatcher thing was removed and replaced by using the ReferenceTrack. I do not see any flicker, and it seems to work nicely on CS/CPP as well. I also added a minor improvement, a new config parameter to keep a minimum gap when extending durations. It always seems a bit harsh to me when two subs follow each other without any gap.

[brabebhin]

I added the changes to MediaSourceConfig. Unfortunately I had quite the merge on the FFmpegMediaSource and in the IDL, can you double check if anything was missed?

We have amassed an impressive number of configuration options over time! I haven't renamed any of the passthrough audio properties yet.

The AutoExtendDuration property is not exposed in the IDL. StreamBufferSize is deprecated in IDL, but we use it once to initiate the avio_context. I wasn't sure if this should really be replaced by ReadAheadBufferSize.

FFmpegOptions - This is an interesting one. It only really works when initiating the FFmpegMediaSource. Maybe we should add an overload to CreateFrom methods to signal its importance? Or create a new set of methods like "CreateFromWithOptions"?

On the topic of compressed audio - winRT has been supporting compressed flac and alac for some time now. Do you think it is worth adding these to the MediaSource?

[lukasf]

Looks good. I done some renaming and threw out all deprecated APIs. There was a bug actually about StreamBufferSize. It was to be replaced by FileStreamReadSize, but it was kept being used. Cleaned up now.

About the audio decoding: I am not sure if it is worth the hassle. PCs don't have any kind of hardware audio decoders. It just does not make sense, given how minor it is on the CPU. It only ever made sense on WP, but that is dead. Maybe Hololens, but seems pretty dead as well. Not sure about Windows on ARM. ARM chips might have hardware decoders (relict from smartphones). Not sure if they are actually used when running on Windows. But it's the only platform alive (more or less) where HW audio decoders might be available at all.

We could consider removing the old properties and replacing with an enum like for the video decoders (Auto/FFmpeg/System). Then we are flexible and could add more formats without modifying the config class each time.

[brabebhin]

We could consider removing the old properties and replacing with an enum like for the video decoders (Auto/FFmpeg/System). Then we are flexible and could add more formats without modifying the config class each time.

Not sure if this is worth the hustle right now. If someone specifically asks for compressed audio samples support I guess we can implement it. This is a feature I personally never used, not even on WP days.

I'd say supporting socket streams has higher priority.

Subs are indeed flicker free on the other branch.

[brabebhin]

Feature wise this looks good. I think we should merge the winui subtitles branch before finally commiting to master.

[brabebhin]

I think we can now remove the DispatcherQueue, we no longer need it.

[lukasf]

True. But we should still check for DispatcherQueue, because I think DisplayInformation::GetForCurrentView (on UWP) will throw if we are not on a Dispatcher thread. But we don't need the DispatcherQueue after that, no need to store it anymore.

[brabebhin]

Do we want to support dynamic format changes for HDR in this PR as well?

[lukasf]

Nah this PR is totally bloated already. Let's not add even more stuff here. Makes it difficult to review as well.

[brabebhin]

Do I need to install some special tool for the desktop build?

It complains it doesn't find openSSL, but as far as I can tell, openSSL does build correctly.

Build_20231219_211015_x86.log

[lukasf]

No, there are no new dependencies. I noticed that the platform seems to be "desktop" instead of "Desktop" in your build. This might cause problems in the scripts (string compares are usually case sensitive). You should try again with "Desktop" and see if that helps. You can also try with -ClearBuildFolders.

[brabebhin]

Nice catch. Desktop does work.

[brabebhin]

Things seem to work nicely. Can't really tell if subtitles flicker now, they seem completely broken for me. No matter what I do with windows accessibility settings. This is windows, not us

[brabebhin]

So I think we're in a bit of a pickle.

In the winui_build we have different packages for the ffmpeg build for desktop or UWP, but we have no overarching one to pick between the 2 automatically. We have the ffmpeginteropx which automatically picks desktop/uwp for both ffmpeginteropx build and ffmpeg build. We need one just for the ffmpeg build.

Now, for 99% of the people this won't be a problem, but it is for us and anyone who wants to use the project in their own SLN: we need to restore a different nuget package based on build config (_desktop or _UWP).

This isn't supported with packages.config.

[lukasf]

I think I just solved this on the ffmpeg-6.1 branch. I just added both the UWP and Desktop FFmpeg nugets to the packages.config. This does not cause problems, it just means that both are restored into the packages folder. From the vcxproj, I only include the right one. Seems to work fine.

[brabebhin]

Ah lol didn't see that yet.

I also tried migrating to packagereference but visual studio behaves strangely with it, even if the build appears to be successful so I dropped it.

It would have been cool if c++ projects supported package reference, we could have used conditions inside the xml to target the right package.

[softworkz]

Maybe rather not spend so much effort on WinUI3. I have come to the conviction that it's dead already:

https://github.com/microsoft/microsoft-ui-xaml/issues?q=is%3Aopen+is%3Aissue+author%3Asoftworkz+sort%3Aupdated-desc+ (you'll see it, didn't want to link directly)

[brabebhin]

Maybe Microsoft just diverted effort to MAUI. That framework was a sorry pile of stinking gue, to the point Microsoft was in danger of losing its entire xamarin/dotnet mobile presence.

Or maybe Microsoft drunk too much of the AI kool aid and has gone insane. Wouldn't be surprised. Maybe they are all cylons.

[softworkz]

Maybe Microsoft just diverted effort to MAUI.

I can't imagine that, because on Windows, MAUI is presenting using WinUI3. And MAUI controls are short on features (well, each one needs to be implemented for all platforms). So it's way more limited than WinUI3 and not a suitable replacement. Also, MAUI always has some "adapter controls" between the logical MAUI control and the native platform control (e.g. WinUI3). That's not a kind of design that is acceptable to be the primary UI framework on Windows.

Ah, now I realize that you didn't mean MAUI to be a replacement for WinUI3, but that they tasked the developers there...

I'm not sure. Pulling all developers away frrom a Windows UI framework to MAUI? I think the priority will always be Windows in general.

I recently mentioned that during Project ReUnion/WinAppSdk, they had realized that they can't support Xbox with WinUI3 (and probably not the other devices like HoloLens or Surface). That was after ReUnion 0.5 and before WinUI3 1.0.

Thinking about it now, that was maybe already the moment from which on WinUI3 has been dead already. But they couldn't stop it anymore as they had made too much progress and starting something new would have lost too much time and would have been emparrassing as well.

I'm wondering what will come next. It cannot be fully .net because the WinRT APIs can't all be re-written. But I could imagine that they are rewriting the xaml ui stuff in .net. Maybe as "WinUI4", maybe even with a kind of migration of xaml code. Or perhaps even with drop-in compatibility.

Personally, I thnik they should just bring WinUI3 features to WPF and make WPF runnable on Xbox and elsewhere.

But when I would have to bet on something, then I'd say: It will be once another XAML flavor which is once again incompatible to all other Xaml flavors ;-)

[brabebhin]

Pretty sure xbox can live without winui and uwp. In fact, xbox existed long before these and will continue to exist with directx and winRT.

As for the desktop, they will likely just fully embrace react native and winRT bindings for it. Not sure where MAUI fits into all this, considering Microsoft themselves are not using it.

WinUI3 was cool cause it allowed xaml controls for native unsandboxed c++. It would have been much easier to just improve the performance of uwp in some areas (like file access) than create a new framework from scratch. I've seen most of the changes made to winui3 came from the winappsdk in the last release.

I need a map with all those abbreviations: uwp, winrt, winappsdk, winui2, winui3 etc Probably microsoft employees also get lost in it.

Oh well...

[softworkz]

Pretty sure xbox can live without winui and uwp. In fact, xbox existed long before these and will continue to exist with directx and winRT.

UWP is the only way to submit apps for Xbox. The other way is with the Xbox game sdk, but you can only create and submit games with it.

As for the desktop, they will likely just fully embrace react native and winRT bindings for it.

That's a very different kind of development model. I mean, yes they do support it and invest in it, but as a primary UI framework? I don't believe that. It would be a littble like giving up their desktop OS at all, saying that everything can be done with htmljs...

[brabebhin]

To be fair, we only use winUI for dealing with the DispatcherQueue and subtitle flickering If WinUI is to go dead, we have a way forward by simply supporting the winRT APIs for subtitles. I actually had several attempts at doing this but to be honest, I was too lazy to do it. But if I gather the necessary chi and patience to do it, i don't think it will take more than 1 week or so of work.

As for HDR support, I long suspected there's a DirectX way of getting it, or something that a desktop app, no CoreWindow can do.

So in that case we would continue to exist as a pure desktop platform + UWP and not depend on winUI at all. I think this PR is improperly named winUI-build now, it is more like a desktop build: we use desktop build for ffmpeg, only ffmpeginteropx has some dependencies on winUI that could be factored out.

It is really quite a shame they just decided to drop UWP and not invest further into polishing it. I really wish I had a sneak peak into those decision making meetings, I have the popcorn prepared already.

As for giving up Windows OS as a whole, that is sort of already happening. The only thing keeping Windows alive right now in the consumer space is the huge market share of the gaming industry, cause otherwise anything you can do on Windows you can do on a Mac or Linux as well. And Microsoft clearly invests first into azure and then into anything else: dotnet is focusing the most on supporting cloud-based workloads, windows 11 OS as a whole hasn't gotten any new interesting and useful features in quite a while (I don't count copilot as useful), so one could say Microsoft = Azure.

[lukasf]

I try not getting too emotional on this topic. I guess we can agree that Microsoft's strategy for Desktop (and Mobile) is a mess, and it has been a mess for years, but it does not help anyone to go on rant about it.

As @brabebhin said, we do not invest much into WinUI, so it wouldn't hurt us much if it gets dumped eventually.

[softworkz]

I try not getting too emotional on this topic.

I am VERY emotional on this topic! 💀 More explicit: I'm p!ssed, since I've just created a WinUI3 application and wasted a lot of time in doing so, not to speak of the PRs I've submitted. Generally I think it's a good decision not to proceed with it, because WinUI3 is way inferior to WPF for example and I don't know many reasons for preferring WinUI3 over WPF other than the sole fact that it's newer.

The other part that's upsetting me is that they are letting people work and struggle with WinUI3, watching developers taking effort in submitting issues and proposals, even though they know already that these will never be acted on. And not just that: they are even trying to create the impression that some actions would be taken: When you look at my issue report, where the guy said he would have forwarded my "status request" to the person who has internally be assigned to the issue. It might not have been strictly a lie, but it definitely created a false impression.

[lukasf]

I try not getting too emotional on this topic.

I am VERY emotional on this topic! 💀

I understand your pain. I've been there (multiple times, actually)...

[brabebhin]

Winui 3 is the 3rd time I rewrite my media player:

1. For windows phone 8.1
2. For windows 10
3. For winui 3 (at this point I kinda gave up anyways and i'm doing it just for keeping my skills sharp)

At this point i'm considering writing it in QT or something lol.

[huynhsontung]

Hello. Are there any blockers for this PR? I compiled and tested this branch with my UWP app. I would love to get this version in a Nuget package. Please let me know if I can help in any way.

[brabebhin]

I'd probably merge the ffmpeg 7 branch first. But we need @lukasf for packages.

[lukasf]

No real blockers, it's just that I have so many other things to do. But I'll try to get things together. ffmpeg 7 branch needs some cleanup before it can get merged. Then we could create some preview packages for testing.

[lukasf]

Wohoo, finally a green CI build, haven't seen that in a long time ^^

[lukasf]

Github actions working fine so far...