Proposal: Enable DynamicDependencies

[DrusTheAxe]

Proposal: Extensibility hooks in generated main

Summary

Optionally add hooks in the generated main code for developers to provide code to execute before any WinUI code. Bonus points for hooks at the end of the generated main and other useful points for developers to add functionality without manually editing or otherwise hacking generated files in fragile ways.

Rationale

• MSIX Dynamic Dependencies requires API calls before using WinUI in a Framework package

MSIX Dynamic Dependencies supports non-packaged processes using Framework package goods by calling APIs at runtime, accomplishing what packaged apps do via <PackageDependency> in their appxmanifest.xml. For non-packaged apps to use XAML via a Framework package the app needs to call APIs before accessing WinUI.

This is currently impossible with XAML's generated main method. There's no opportunity for the developer to provide code to execute before the generated main uses XAML (not without fragile manual editing of the generated main). Per https://github.com/microsoft/ProjectReunion/issues/89#issuecomment-701455098

Developers shouldn't have to write code in their main method. Maybe this is fine for frameworks where the app author owns the main method. But in WPF and WinUI, the main method is generated by tooling.

The generated main should have extensibility hooks, e.g. the generated main has pre() and post() calls to InsertYourCodeHere? Maybe a new main-extensions.cpp is created with a new XAML project with void XamlMainPre() { /insert code here/ } and XamlMainPost() for devs to inject logic into their app's startup flow?

There may be other reasons (e.g. a crash handler).

See Proposal: Extensibility hooks in generated main #3632 for the equivalent ask of WPF.

Scope

Capability	Priority
Optionally support a 'pre-Main' callout for developers to provide code before the generated main logic w/o editing generated files	Must
Optionally support a 'post-Main' callout for developers to provide code after the generated main logic w/o editing generated files	Should

Open Questions

Are there other desirable extension points in generated files/code?

[StephenLPeters]

Can you explain the scenario in which you need this? It will help the team understand the requirements better and prioritize it against our other work.

[StephenLPeters]

The App constructor happens very early on, that is a potential place for you to put the code you want to run, is it not?

[DrusTheAxe]

Can you explain the scenario in which you need this? It will help the team understand the requirements better and prioritize it against our other work.

It's required to use Project Reunion.

Non-packaged processes must call MddBootstrapInitialize() before they can access any resources in the Project Reunion Framework package including the Dynamic Dependencies API, and they must call the Dynamic Dependencies API before accessing any resource in any other framework package.

MddBootstrapInitialize and Dynamic Dependencies is required by non-packaged processes to use resources in Framework packages as, unlike packaged apps, they have no appxmanifest.xml to declare a <PackageDependency>. MddBootstrapInitalize and Dynamic Dependencies APIs are a runtime equivalent to <PackageDependency>.

LoadLibrary() and ActivateInstance() fail to find and load code in the Project Reunion Framework package if called before MddBootstrapInitialize(), and fail to find or load code in other Framework packages if called before the Dynamic Dependencies API is called.

Where can a XAML app make these calls, before any XAML API is called?

Boring details: DynDep API provided via a DLL in a Project Reunion framework package but non-packaged processes can't use a framework (w/o severe problems). The 'bootstrapper API' is the solution that that chicken/egg problem. MddBootstrapInitialize() finds the appropriate Project Reunion framework and makes it available to the current process. If all you need is the Project Reunion framework package then you're done. If you need other frameworks you're now free to call the DynDep API to make them available too.

[DrusTheAxe]

@StephenLPeters The App constructor happens very early on, that is a potential place for you to put the code you want to run, is it not?

Please define 'very early'? Does this happen before ALL calls to WinUI?

MddBootstrapInitialize and Dynamic Dependencies APIs must be called before any attempt to LoadLibary(winui.dll), ActivateInstance(microsoft.ui.xaml.whatever), etc

[StephenLPeters]

No, we will have loaded winui before the app constructor is called. Seems like we'd need to implement this for your scenario.

[StephenLPeters]

@fabiant3 this is a tooling proposal, is there a PM in this space we can assign it to?

[fabiant3]

Adding @marb2000

[JeanRoca]

@marb2000 could you let us know who in Project Reunion would be on point for this?

[marb2000]

@JeanRoca you can redirect all these request towards me. @DrusTheAxe, My question is why can't we do this in the Main before loading the App object? If so, this is about changing the codegen.

[DrusTheAxe]

The needed code pattern is

main()
{
    MddBootstrapInit(...);
    ...use WinUI...

Is WinUI touched before this? This includes global constructors etc

Put overly simplistically, think of MddBootstrapInit() as

string path = GetEnvironmentVariable("PATH")
path = "C:\Program Files\WindowsApps\WinUI_1.2.3.4_x64__1234567890abc"  + ";" + path
SetEnvionrmentVariable("PATH", path)

WinUI's DLLs can't be accessed before MddBootstrapInit() is called.

It's my understanding (per folks more expert in XAML :P) the generated code was invoked before the developer had the opportunity to do this, hence the issue. Are they mistaken?

[marb2000]

This is the generated code:

public static class Program
    {
        [global::System.CodeDom.Compiler.GeneratedCodeAttribute("Microsoft.UI.Xaml.Markup.Compiler"," 0.0.0.0")]
        [global::System.Diagnostics.DebuggerNonUserCodeAttribute()]
        [global::System.STAThreadAttribute]
        static void Main(string[] args)
        {
            global::WinRT.ComWrappersSupport.InitializeComWrappers();
            global::Microsoft.UI.Xaml.Application.Start((p) => {
                var context = new global::Microsoft.System.DispatcherQueueSynchronizationContext(global::Microsoft.System.DispatcherQueue.GetForCurrentThread());
                global::System.Threading.SynchronizationContext.SetSynchronizationContext(context);
                new App();
            });
        }
    }

So If the request is to add MddBootstrapInit(...); in the generated code before Application.Start, I don't see why we can't do it. I believe we want to use FrameworkPackages by default. However, if the request is to enable an extensibility point where every developer can add things before XAML starts easily, then it's more complicated and we need to design it. Also, devs can create their own Main and do whatever they want before XAML loads.

Adding @stevenbrix to the discussion.

[DrusTheAxe]

MddBootstrap.h

STDAPI MddBootstrapInitialize(
    const PACKAGE_VERSION minVersion) noexcept;

STDAPI_(void) MddBootstrapShutdown() noexcept;

That minVersion is how an app specifies "I need Project Reunion, version 1.2.3.4+". Nothing something codegen can treat as a constant. I guess devs could specify a new parameter to the generator and have it emit the call, and rerun the generator to update it whenever the dev moves their minimum-required-version. But seems kinda heavyweight.

Also, this is only needed by not-packaged apps. Packaged apps using Project Reunion don't need this call (in fact it'll fail for them). The bootstrapper is to help not-packaged processed get a toeheld in the packaged world so they can access things formerly not accessible, e.g. framework packages.

Thoughts?

BTW that's C#. What's the generator emit for C++?

[marb2000]

I'm just brainstorming here...

If the way that a developer specify a new version of Reunion is updating the Reunion NuGet, can the toolchain autogenerate the code again?

[DrusTheAxe]

I'd shy away from any nuget-dependent magic solutions. We're initially supporting nuget but there's been asks for other formats e.g. vcpkg.

And not every developer uses VS or nuget. Arguably they'd crack open the nuget and handle the files as needed in their build system. Any XAML codegen magic based on nuget wouldn't be viable in these cases and manually editing generated files is feasible but probably more fragile than desirable.

In earlier conversations some thought there's merit in having a hook to meet this need via a generic mechanism. That devs could use for other purposes (the notion of pre- and post- hooks are as age old concepts), and it's up to them to stick the MddBootstrapInit and/or whatever in a non-generated file.

I just learned about __has_include. Might that be a simple solution e.g. update the generated code to be

#if __has_include(xaml_main_hooks.h)
#  include xaml_main_hooks.h
#endif

int WinMain(...)
{
#if __has_include(xaml_main_hooks.h)
    Xaml::Hooks::Pre();
#endif

    ...current generated code...

#if __has_include(xaml_main_hooks.h)
    Xaml::Hooks::Post();
#endif

    return 0;
}

Devs can create xaml_main_hooks.h with a Pre() function calling MddBootstrapInit (and/or whatever) And if devs don't care they don't define a xaml_main_hooks.h and XAML's codegen ignores it

At least, for C++. For C# if nothing else there's the more traditional #if XAML_MAIN_HOOKS sort of test.

[sylveon]

I feel like anything relying on the include path could be recipe for potential trouble and hard to diagnose issues, because of libraries "leaking" implementation details like these in the include path of another project/library.

And I absolutely agree with not making it rely on NuGet: NuGet for C++ dependencies is a complete pain and I'd much rather use vcpkg for everything I can, but projects like WinUI and cppwinrt sticking to NuGet only prevent me from switching over.

[stevenbrix]

It's sounding like we'll have to have different solutions for .NET and C++ customers.

If the way that a developer specify a new version of Reunion is updating the Reunion NuGet, can the toolchain autogenerate the code again?

Yes, this is possible. For .NET customers, Nuget is the only way to go, and we should use a Source Generator for this, so that we can use a single implementation for WPF and WinUI customers.

And I absolutely agree with not making it rely on NuGet: NuGet for C++ dependencies is a complete pain and I'd much rather use vcpkg for everything I can, but projects like WinUI and cppwinrt sticking to NuGet only prevent me from switching over.

@Scottj1s, is Reunion as a whole looking at supporting vcpkg? I thought vcpkg had limitations that prevented us from migrating?

[DrusTheAxe]

@stevenbrix is Reunion as a whole looking at supporting vcpkg?

Project Reunion is targeting NuGet for the initial release. Other distribution mechanisms (including vcpkg) have been discussed for potential future consideration. See CMake or vcpkg support #74 for more details

[sylveon]

Even if vcpkg isn't meant to be supported for the first release, I think it would be wise to design a solution that is not entirely coupled to NuGet to ensure that when vcpkg gets support it doesn't break existing consumers using NuGet.

[DrusTheAxe]

wise to design a solution that is not entirely coupled to NuGet +1

We've had interest in other technologies e.g. CMAKE. Consider NuGet ***a*** distribution mechanism (not the). Merely the 1st we're supporting.

Likewise, the solution should support C#,. not just .NET Core. .NET 4.x is quite popular and not all devs can or will switch to .NET Core anytime soon.

The generator-hook concept is one option but if there's better answers that's fine by me. I have no dog in this hunt other than "a good solution for all". I defer to the experts on its shape and size :-)

[stevenbrix]

+1 to a solution for all.

I'm going to change the title of this issue, because the problem is that we need to enable dynamic dependencies, and extensibility hooks are one possible solution.

[stevenbrix]

Also, just so we're speaking correctly - nothing we do as part of the build is "based on nuget" - NuGet is just our delivery mechanism. We do depend on MSBuild, and I don't see that changing any time soon. My understanding is that CMake is a layer on-top of MSBuild, so I don't see our MSBuild dependency being problematic. If my understanding is wrong, then please let me know.

Another possible solution is that we generate the boilerplate code needed to call this API and register the framework packages. We have all the necessary information at compile time to do this, so my vote would be to not require more boilerplate if we don't absolutely have to. The same thing applies for sparse packages. We have tooling, which we hope to preview soon, which will generate the sparse package for you in VS, so all the developer has to do is write the boilerplate code to register it. Since the code to register it is only changes based off the name of the package, we can generate the proper code at compile time as well.

[sylveon]

My understanding is that CMake is a layer on-top of MSBuild, so I don't see our MSBuild dependency being problematic. If my understanding is wrong, then please let me know.

Not entirely. CMake supports multiple generators, one of them being a MSBuild generator. Many open source projects however, target the Ninja generator, in which case MSBuild is not part of the build process.

[DrusTheAxe]

@stevenbrix generate the boilerplate code needed to call this API and register the framework packages

No registration. ProjectReunion handles that.

I don't see how reliance on any particular tool (NuGet, MSBuild, CMake, Ninja, GNU Make, SCons, VS, VSCode, Emacs, vi, yadda yadda) is viable as Project Reunion isn't dictating or ruling out any (and all) tools. Quite the opposite. Given the breadth of development processes and practices (including many one-off custom) we aim to support them all (though some maybe sooner or better than others :-)

To recap: a non-packaged process using WinUI 3 needs to call MddBootstrapInitialize(...parameters...) before touching any WinUI bits, anything else from Project Reunion or any other framework package. And it's not a fixed/constant call. Those parameters can vary per-app, and we reserve the right to extend them in the future :-)

[riverar]

Is there any additional thinking/progress being made on this? It doesn't work today (0.8-preview) and I'm getting very worried this won't work at 1.0 GA.

Please also consider apps that don't use xaml but want to consume WinUI components. For example, I'm using WinUI via Rust and bootstrapping appropriately but WinUI blows up.

[mqudsi]

FWIW, not mentioned in this issue is that the generated Main() entrypoint for WinUI projects is gated behind an #if !DISABLE_XAML_GENERATED_MAIN preprocessor check. To work around some Windows App SDK and WinUI threading model bugs pertaining to background activation triggering RPC_E_WRONG_THREAD exceptions, I too needed to inject some code into Main() prior to anything touching the WinUI or WindowsAppSDK libraries, and I was able to do so without too much difficulty thanks to that preprocessor check.

If you just define DISABLE_XAML_GENERATED_MAIN in your project and then add a Program.cs that includes the following, you get a non-generated equivalent to the default startup code that you can then modify to inject any init calls prior to WinUI/WindowsAppSDK calls:

namespace Foo
{
#if DISABLE_XAML_GENERATED_MAIN
    public static class Program
    {
        [STAThread]
        static void Main(string[] args)
        {
            global::WinRT.ComWrappersSupport.InitializeComWrappers();
            global::Microsoft.UI.Xaml.Application.Start((p) => {
                var context = new global::Microsoft.System.DispatcherQueueSynchronizationContext(global::Microsoft.System.DispatcherQueue.GetForCurrentThread());
                global::System.Threading.SynchronizationContext.SetSynchronizationContext(context);
                new App();
            });
        }
    }
#endif
}

In terms of directing developers developing unpackaged apps to do this themselves (vs some sort of pre/post hooks), it's not too bad (the ugliest part of it is defining the constant in the csproj file), but my concern right out of the gate was that there's no guarantee that any of the code I copy-and-pasted out of the generated Program.cs won't change at any time; say with WinUI 3.1 the devs decide to add another call after InitializeComWrappers() - I'll have no clue and my alternate Main() will continue to be called and run as-is, but might run into undefined behavior at runtime because it skipped doing something integral that would have been included in an automatically updated, generated Program.cs.

---

Can I ask why Program.cs is generated rather than just being part of the WinUI 3 project template? Back in the days of C# 1.0, we had a Program.cs included as part of the "C# Windows Forms" template, and it really didn't cause any grief. Developers just getting started simply ignored it and focused on the main window SWF C# file, but when you needed to dig into the application startup code (say to add single-instance detection and redirection), the Program.cs file with its Main() routine was right there for you to hack away at.

To give a more modern and perhaps more relevant example, ASP.NET Core projects again include a Program.cs file with the Main() startup routine as part of the template rather than generated as part of the build process. In the upgrade from ASP.NET Core 2.2 to ASP.NET Core 3.1, breaking changes were made to the ASP.NET Core entrypoint that required users to manually open and patch the Program.cs file to start up the ASP.NET Core runtime in the new fashion and it was simply documented in the release notes.*

* To be fair, the old startup code wouldn't compile - which is a good thing. I'd go so far as to include a hard-coded version number of version enum as a trailing parameter e.g. Microsoft.UI.Xaml.Application.Start((ApplicationInitializationCallback) ..., ApplicationStartupVersion.Version3) and throw an exception at startup in a subsequent release that requires a different initialization procedure with a helpful error message and maybe a link to the release notes/upgrade docs so that no one calls into a newer version of the Application startup with an older version of the startup code.

Or you could include a non-generated partial static class Program in a Program.cs in the template and in the Main() just call into a generated Program.generated.cs's partial static class Program { void StartApp() { ... } } and move all init code into there; it would be always regenerated so always up to date, but developers would have access to the non-generated Main() in the non-generated Program.cs to do whatever they want with.

Personally, I'd go with option a over option b because I prefer less voodoo and there's already way too much of that when dealing with packaged apps and WinRT.

[DrusTheAxe]

WinUI3 support was enabled via WinAppSDK's Bootstrap auto-initializer.

If this sort of Wi UI extensibility is of interest for other reasons please open a new issue and link to this for historical reference

[sylveon]

I don't understand how this was silently renamed from "extensibility hooks in main" to "enable dynamic dependencies". This feels like an escape to close the issue for me.

The need to run your own code before App's constructor is still very much present, and hasn't been addressed. For example, I want to set DLL loading policies and setup my logging framework before doing any WinRT work, as those affect the WinRT runtime and loading of runtime objects (logging restricted error info, forbidding unsigned DLLs from loading, forcing ASLR and CFG, etc.)

[mqudsi]

It’s not a very good option from a maintainability perspective because you have to manually backport improvements or changes back to your code, but in lieu of a sensible solution here you can use DISABLE_XAML_GENERATED_MAIN to call your own version of the same code (copy-and-pasted from the autogenerated one) with your callbacks/hooks in place.

I don’t disagree that MS should provide a better option, of course.

Edit: sorry, it’s been so long I forgot/didn’t realize I already suggested this.

[DrusTheAxe]

@stevenbrix renamed it, I don't entirely know why. I opened the issue as we needed a way to support process setup work before any WinUI WinRT object was activated and through a more general extensibility mechanism would be simpler and more broadly useful. But folks disagreed and ultimately we needed Dynamic Dependencies to work regardless of broader WinUI interests (met or not) and thus was born the WinAppSDK Bootstrap auto-initializer.

Given the driving reason for this issue was WinUI support for WinAppSDK via Dynamic Dependencies the rename isn't necessarily wrong. But as noted on the thread there are more uses for such open extensibility than just Dynamic Dependencies. If that's of interest to the community I suggest opening a new issue with the details, as clearly "open WinUI generated main to support DynDep and other uses" didn't address those other uses.

I'm supportive of the broader extensibility options and reasons raised on this thread. And if/when WinUI evolves to support them (or other useful mechanisms) I'll happily revisit Dynamic Dependencies (and other) support by WinUI.

• @MikeHillberg @Scottj1s for visibility