Add support for Windows App SDK instantiation

[jhancock4d]

Would it be possible to create a sample of using project reunion with this and how to deploy it with a flutter app?

The reason why this is important is that without project reunion creating things like full featured video players with DRM, OpenIdConnect authentication windows and the like is incredibly difficult with C++ (you have to get down into the Media Foundation layer which is brutally complex as an example) or downright impossible because it isn't accessible from win32.

With this project and a project reunion sample, the sky's the limit and we can quickly backfill all of the existing functionality and reach parity with Android and iOS.

Thanks!

[timsneath]

I don't fully understand the question, I'm afraid, primarily because I still don't understand Project Reunion.

From reading their docs, it seems to encompass a lot of things: WinUI, a rewrite of DWrite, some tooling, some rather vague comments about unifying UWP and Win32, and some notion of delivering APIs via NuGet. It's not clear what it would mean to "use Project Reunion" from this package. I'm very unclear right now about what I'd change to use Project Reunion at this time.

Can you enlighten me? Is there a sample or doc that demonstrates what you're trying to do from Dart today?

Sorry if this is a dumb question -- even after many years' experience with Microsoft Windows development, mostly from the inside, their naming causes a lot of confusion.

[jhancock4d]

@timsneath You're not alone. Basically UWP is dead except perhaps on Hololens and XBox and even that isn't clear if that will survive.

The future is WinUI + Project Reunion. WinUI is an extension of WPF for the most part although it shares a lot with UWP.

The key to project reunion nuget libraries, which can be used in C++ and .NET is that they can be used outside the UWP sandbox so WinRT is no longer a thing persay.

This is important to flutter for many reasons: 1. UWP is dead. 2. To be able to create a win32 app that for example does DRM Video with captioning etc. or OpenIdConnect protected authentication, you must use Project reunion endpoints (is my understanding) There are countless other things like Windows Hello, sensors and others that can't happen without Reunion.

So you're work on WinRT is absolutely appropriate, but I'd suggest that it should be focused on getting Reunion libraries working, not WinRT which is pretty much a dead duck as a directly accessible set of APIs from my understanding.

Here's some C++ samples I found: https://github.com/megayuchi/Project-Reunion-Samples/tree/main/HelloWorld/reunioncppdesktopsampleapp

I'd also guess that there will be cases where Flutter plugins will need access to Win UI 3 too.

[timsneath]

This is super helpful -- thanks.

[gitslav]

This should be renamed to "Windows App SDK Sample". https://docs.microsoft.com/en-us/windows/apps/windows-app-sdk/

[timsneath]

At the moment, this seems relatively limited in value. The majority of modern Windows APIs are designed as Windows Runtime APIs. Some new APIs seem to be packaged in the Windows App SDK for out-of-band distribution (e.g. WinUI 3). Regardless, these APIs use the Windows Runtime ABI and component model, so getting a solid Windows Runtime implementation is a prerequisite to any support for Windows App SDK features.

[jhancock4d]

@timsneath The primary thing that I think is of concern would be on video playback and specifically with DRM. Is this doable without Windows App SDK?

[timsneath]

I don't know that I'd start with this package to achieve that. At least at the moment, I think this would involve deeper orchestration with the engine, so it would likely make sense as a shared texture in the same way as webviews or platform views. That is, the code would likely be written in C++ anyway.

[jhancock4d]

@timsneath It's my understanding that these APIs are only available in RT apps OR Windows App SDK. Is this incorrect?

[timsneath]

I don't see anything in Windows App SDK that focuses on video, actually. https://docs.microsoft.com/en-us/windows/apps/windows-app-sdk/

But there are plenty of APIs for video playback in Windows, for example Media Foundation: https://docs.microsoft.com/en-us/windows/win32/medfound/how-to-play-protected-media-files

[jhancock4d]

Ok, well WinRT video is easy including DRM. Media Foundation is so complex that the people in Microsoft don't even understand it fully and virtually no one uses it directly. If Flutter waits for someone to implement media foundation video with DRM Windows will never reach pairity.

I was hoping you were going to say that you can access the RT equivalent APIs directly without having to do this work. (i.e. in C# UWP it's 4 lines of code to do PlayReady video playback. Same for AES encrypted video and like 3 more lines for closed captioning.

[timsneath]

Sorry, I'm confused: which APIs are you talking about? Again, I don't see anything in Windows App SDK for video rendering. Perhaps you can point me at the APIs that you're thinking we should support with the win32 package?

Flutter can reach anything in Windows. There are no boundaries: a Flutter app on Windows is a Windows app.

[jhancock4d]

Basically, all of this: https://docs.microsoft.com/en-us/uwp/api/windows.media.playback.mediaplayer?view=winrt-22000

Which I believe is now part of WinUI 3 functionality and thus the SDK.

Windows.Media.Protection is DRM and Timed Track stuff is the closed captioning.

And then on the other side is streaming.adaptive and .transcoding for video recording. There's other important stuff like speech synthesis in there too, but that's a start.

If you were to do this from the low level media foundation stuff, the task is overwhelming. But to build the functionality in flutter from these higher level apis is pretty trivial and will encourage development and thus parity.

If you're saying that we can already get to these, that's very cool. If you're saying that all of the video stuff has to be done with Media Foundation, that's self-destructive. There's a reason why virtually no one has even attempted to create a Windows Media Center clone with DRM in it. (Even Plex, the most commercially successful of the clones.) This is THE reason. It's brutal, and if you don't use these APIs then you have to go through $50k certification, so you'll never get apps like Netflix on Flutter unless someone like Google or Netflix pays for the certification, and because of the cost I'd almost guarantee that work won't be open sourced and it isn't obvious that even if it was, that that would certify all apps using it as using these APIs do. (i.e. you're certified PlayReady if you're code uses these APIs and thus can write software like DVRs etc.)

[timsneath]

I don't believe there's any relationship between this and WinUI 3 -- it's a Windows Runtime API.

But again, if someone were to implement a media player using this, I wouldn't recommend they start with the Win32 package. They'd write a C++ plugin that would call any of these APIs and integrate with Flutter at a lower level (via a shared texture).