MSIX Dynamic Dependencies - allow any process to use MSIX Framework packages

[DrusTheAxe]

Proposal: MSIX Dynamic Dependencies (aka DynamicDependencies aka DynDep)

Summary

Provide APIs to enable access to packaged content at runtime, regardless if the caller is packaged or not. This supplements the MSIX appmodel's current static dependency support (via in appxmanifest.xml) with a dynamic runtime equivalent. It also allows non-packaged processes (which have no appxmanifest.xml) to use packaged content.

Microsoft-internal task 23447728

Rationale

• Enable all apps (MSIX-packaged and not) to use framework packages
• Support dynamic selection of framework packages at runtime to complement static selection of framework packages at development/install time (via appxmanifest.xml) DynamicDependencies is the moral equivalent of LoadLibrary - DynamicDependencies supplements static (manifest'd) dependencies as LoadLibrary supplements EXE/DLL import references.

This aligns with Project Reunion's roadmap:

• Foundational technology enabling new Project Reunion features to be delivered via Framework packages but accessible by all applications (packaged and not-packaged)
• Enable all applications (packaged and not-packaged) to use WinUI's Framework package

Scope

Capability	Priority
APIs available to MSIX and non-MSIX applications	Must
APIs to add and remove a package dependency dynamically at runtime	Must
API to prevent removal of packages not in use by running processes (i.e. install-time pinning)	Must
APIs to enumerate package dependencies in use	Could
APIs to enumerate defined package dependencies (install-time pinned)	Could
Win32 API	Must
WinRT API	Should

Important Notes

All processes have a package graph. A process may be created with entries in its package graph; this is referred to as the 'static package graph'.

Packaged processes (i.e. a process with package identity) are created with a static package graph per their AppxManifest.xml. A process' static package graph cannot be altered, but it can be supplemented at runtime via the Dynamic Dependency API.

Processes without package identity have an no static package graph. They can modify their package graph using the Dynamic Dependency API.

MddPinPackageDependency defines a package dependency.

MddAddPackageDependency determines a package that satisfies a package dependency and updates the caller's process. This includes adding the resolved package to the process' package graph, updating the Loader to include the resolved package in the DLL Search Order, etc. The package dependency is resolved to a specific package if not already resolved.

A resolved PackageDependency is represented by MDD_PACKAGE_DEPENDENCY_CONTEXT.

Once a PackageDependency is resolved to a package all further MddAddPackageDependency calls yield the same result until the package dependency is unresolved. Resolved package dependencies are tracked by User + PackageDependencyId. This ensures multiple overlapping calls to MddAddPackageDependency yield the same result. A package dependency is unresolved when the last MDD_PACKAGE_DEPENDENCY_CONTEXT is closed (via MddRemovePackageDependency or process termination).

MddRemovePackageDependency removes the resolved PackageDependency from the calling process' package graph.

MddUnpinPackageDependency undefines a package dependency previously defined via PinPackageDependency.

PackageDependency definitions and usage are tracked and managed on a per-user basis.

PackageDependency definitions are not persisted or tracked across reboots if MddPinPackageDependency is called with MddPinPackageDependency_LifecycleHint_Process. Specify MddPinPackageDependency_LifecycleHint_FileOrPath or MddPinPackageDependency_LifecycleHint_RegistrySubkey for PinPackageDependency to persist the definition until explicitly removed via MddUnpinPackageDependency or the specified lifetime artifact is deleted.

If concurrent processes need the same package resolution for a defined criteria they should share the packageDependencyId returned by MddPinPackageDependency. Concurrent processes running as the same user calling MddAddPackageDependency with the same packageDependencyId get the same resolved package added to their package graph. This enables multiple concurrent processes needing the same package resolution get a consistent answer.

Package dependencies can only be resolved to packages registered for a user. As packages cannot be registered for LocalSystem the Dynamic Dependencies feature is not available to callers running as LocalSystem.

Win32 API

MsixDynamicDependency.hpp

enum class MddPinPackageDependency : uint32_t
{
    MddPinPackageDependency_None                               0,
    MddPinPackageDependency_DoNotVerifyDependencyResolution    0x00000001,
    MddPinPackageDependency_LifecycleHint_Process              0,
    MddPinPackageDependency_LifecycleHint_FileOrPath           0x00000002,
    MddPinPackageDependency_LifecycleHint_RegistrySubkey       0x00000004,

     // Define the package dependency for the system, accessible to all users
     // (default is the package dependency is defined for a specific user).
     // This option requires the caller has adminitrative privileges.
    MddPinPackageDependency_ScopeIsSystem                      0x00000008,
};
DEFINE_ENUM_FLAG_OPERATORS(MddPinPackageDependency)

enum class MddAddPackageDependency : uint32_t
{
    MddAddPackageDependency_None                       0,
    MddAddPackageDependency_OnlyUseFirstPackageFamily  0x00000001,
    MddAddPackageDependency_PrependIfRankCollision     0x00000002,
};
DEFINE_ENUM_FLAG_OPERATORS(MddAddPackageDependency)

#define PACKAGE_DEPENDENCY_RANK_DEFAULT 0

enum class MddPackageDependencyProcessorArchitectures : uint32_t
{
    MddPackageDependencyProcessorArchitectures_None = 0,
    MddPackageDependencyProcessorArchitectures_Neutral = 0x00000001,
    MddPackageDependencyProcessorArchitectures_X86     = 0x00000002,
    MddPackageDependencyProcessorArchitectures_X64     = 0x00000004,
    MddPackageDependencyProcessorArchitectures_Arm     = 0x00000008,
    MddPackageDependencyProcessorArchitectures_Arm64   = 0x00000010,
    MddPackageDependencyProcessorArchitectures_X86A64  = 0x00000020,
};
DEFINE_ENUM_FLAG_OPERATORS(MddPackageDependencyProcessorArchitectures)

DECLARE_HANDLE(MDD_PACKAGEDEPENDENCY_CONTEXT);

// Define a package dependency expressing a relationship between an
// application and a packaged component or application. The criteria
// for a PackageDependency (package family name, minimum version, etc)
// may match multiple packages, but ensures Deployment won't remove
// a package if it's the only one satisfying the PackageDependency.
//
// @note A package matching a PackageDependency pin can still be removed
//       as long as there's another that satisfies the PackageDependency.
//       For example, if Fwk-v1 is installed and a PackageDependency specifies
//       MinVersion=1 and then Fwk-v2 is installed, Deployment could remove
//       Fwk-v1 because Fwk-v2 will satisfy the PackageDependency. After Fwk-v1
//       is removed Deployment won't remove Fwk-v2 because it's the only package
//       satisfying the PackageDependency. Thus  Fwk-v1 and Fwk-v2 (and any other
//       package matching the PackageDependency) are 'loosely pinned' – Deployment
//       guarantees it won't remove a package if it would make a PackageDependency
//       unsatisfied.
//
// A PackageDependency specifies criteria (package family, minimum version, etc)
// and not a specific package. Deployment reserves the right to use a different
// package (e.g. higher version) to satisfy the PackageDependency if/when
// one becomes available.
//
// @param user the user scope of the package dependency. If NULL the caller's
//        user context is used. MUST be NULL if MddPinPackageDependency_ScopeIsSystem
//        is specified
// @param lifetimeArtifact MUST be NULL if MddPinPackageDependency_LifecycleHint_Process (default)
// @param packageDependencyId allocated via LocalAlloc; use LocalFree to deallocate
//
// @note MddPinPackageDependency() fails if the PackageDependency cannot be resolved to a specific
//       package. This package resolution check is skipped if
//       MddPinPackageDependency_DoNotVerifyDependencyResolution is specified. This is useful
//       for installers running as user contexts other than the target user (e.g. installers
//       running as LocalSystem).
STDAPI MddPinPackageDependency(
    PSID user,
    _In_ PCWSTR packageFamilyName,
    PACKAGE_VERSION minVersion,
    MddPackageDependencyProcessorArchitectures packageDependencyProcessorArchitectures,
    PCWSTR lifetimeArtifact,
    MddPinPackageDependency flags,
    _Outptr_result_maybenull_ PWSTR* packageDependencyId);

// Undefine a package dependency. Removing a pin on a PackageDependency is typically done at uninstall-time.
// This implicitly occurs if the package dependency's 'lifetime artifact' (specified via MddPinPackageDependency)
// is deleted. Packages that are not referenced by other packages and have no pins are elegible to be removed.
//
// @warn MddUnpinPackageDependency() requires the caller have administrative privileges
//       if the package dependency was pinned with MddPinPackageDependency_ScopeIsSystem.
STDAPI MddUnpinPackageDependency(
    _In_ PCWSTR packageDependencyId);

// Resolve a previously-pinned PackageDependency to a specific package and
// add it to the invoking process' package graph. Once the dependency has
// been added other code-loading methods (LoadLibrary, CoCreateInstance, etc)
// can find the binaries in the resolved package.
//
// Package resoution is specific to a user and can return different values
// for different users on a system.
//
// Each successful MddAddPackageDependency() adds the resolve packaged to the
// calling process' package graph, even if already present. There is no
// duplicate 'detection' or 'filtering' applied by the API (multiple
// references from a package is not harmful). Once resolution is complete
// the package stays resolved for that user until the last reference across
// all processes for that user is removed via MddRemovePackageDependency (or
// process termination).
//
// MddAddPackageDependency() adds the resolved package to the caller's package graph,
// per the rank specified. A process' package graph is a list of packages sorted by
// rank in ascending order (-infinity…0…+infinity). If package(s) are present in the
// package graph with the same rank as the call to MddAddPackageDependency the resolved
// package is (by default) added after others of the same rank. To add a package
// before others o the same rank, specify MddAddPackageDependency_PrependIfRankCollision.
//
// Every MddAddPackageDependency should be balanced by a MddRemovePackageDependency
// to remove the entry from the package graph. If the process terminates all package
// references are removed, but any pins stay behind.
//
// MddAddPackageDependency adds the resulting package to the process' package
// graph, per the rank and options/flags parameters. The process' package
// graph is used to search for DLLs (per Dynamic-Link Library Search Order),
// WinRT objects and other resources; the caller can now load DLLs, activate
// WinRT objects and use other resources from the framework package until
// MddRemovePackageDependency is called. The packageDependencyId parameter
// must match a package dependency defined for the calling user or the
// system (i.e. pinned with MddPinPackageDependency_ScopeIsSystem) else
// an error is returned.
//
// @param packageDependencyContext valid until passed to MddRemovePackageDependency()
// @param packageFullName allocated via LocalAlloc; use LocalFree to deallocate
STDAPI MddAddPackageDependency(
    _In_ PCWSTR packageDependencyId,
    INT32 rank,
    MddMddAddPackageDependency flags,
    _Out_ MDD_PACKAGEDEPENDENCY_CONTEXT* packageDependencyContext,
    _Outptr_opt_result_maybenull_ PWSTR* packageFullName);

// Remove a resolved PackageDependency from the current process' package graph
// (i.e. undo MddAddPackageDependency). Used at runtime (i.e. the moral equivalent
// of FreeLibrary).
//
// @note This does not unload loaded resources (DLLs etc). After removing
//        a package dependency any files loaded from the package can continue
//        to be used; future file resolution will fail to see the removed
//        package dependency.
STDAPI MddRemovePackageDependency(
    _In_ MDD_PACKAGEDEPENDENCY_CONTEXT packageDependencyContext);

// Return the package full name that would be used if the
// PackageDependency were to be resolved. Does not add the
// package to the process graph.
//
// @param packageFullName allocated via LocalAlloc; use LocalFree to deallocate
STDAPI MddGetResolvedPackageFullNameForPackageDependency(
    _In_ PCWSTR packageDependencyId,
    _Outptr_result_maybenull_ PWSTR* packageFullName);

// Return TRUE if packageDependencyId1 and packageDependencyID2
// are associated with the same resolved package.
STDAPI_(BOOL) MddArePackageDependencyIdsEquivalent(
    _In_ PCWSTR packageDependencyId1,
    _In_ PCWSTR packageDependencyId2);

// Return TRUE if PackageDependency pins would produce the same package
// when resolved e.g. whether they share the same packageFamilyName,
// minVersion, and packageDependencyProcessorArchitectures values.
STDAPI_(BOOL) MddArePackageDependencyContextsEquivalent(
    _In_ const PACKAGEDEPENDENCY_CONTEXT* packageDependencyContext1,
    _In_ const PACKAGEDEPENDENCY_CONTEXT* packageDependencyContext2);

NOTE: All APIs prefixed with Mdd/MDD for MSIX Dynamic Dependencies.

WinRI API

namespace Microsoft.ApplicationModel
{
enum PackageDependencyProcessorArchitectures
{
    None    = 0,
    Neutral = 0x00000001,
    X86     = 0x00000002,
    X64     = 0x00000004,
    Arm     = 0x00000008,
    Arm64   = 0x00000010,
    X86A64  = 0x00000020,
};

runtimeclass PinPackageDependencyOptions
{
    PinPackageDependencyOptions();

    PackageDependencyProcessorArchitectures Architectures;
    boolean DoNotVerifyDependencyResolution;
    String lifetimeArtifactFileOrPath;
    String lifetimeArtifactRegistrySubkey;
}

runtimeclass AddPackageDependencyOptions
{
    AddPackageDependencyOptions();

    int Rank;
    boolean OnlyUseFirstPackageFamily;
    boolean PrependIfRankCollision;
}

runtimeclass PackageDependency
{
    PackageDependency(String id);

    String Id { get; }

    static boolean AreEquivalent(String packageDependencyId1, String packageDependencyId2);

    boolean AreEquivalent(PackageDependency otherPackageDependency);

    static PackageDependency Pin(
        String packageFamilyName,
        PackageVersion minVersion,
        PinPackageDependencyOptions options);

    static PackageDependency PinForUser(
        Windows.System.User user,
        String packageFamilyName,
        PackageVersion minVersion,
        PinPackageDependencyOptions options);

    static PackageDependency PinForSystem(
        String packageFamilyName,
        PackageVersion minVersion,
        PinPackageDependencyOptions options);

    void Unpin();

    PackageDependencyContext Add();
    PackageDependencyContext Add(AddPackageDependencyOptions options);

    void Remove();
}

struct PackageDependencyContextId
{
    UInt64 Id;
};

runtimeclass PackageDependencyContext : ICloseable
{
    PackageDependencyContext(PackageDependencyContextId);   //Parameter is MDD_PACKAGEDEPENDENCY_CONTEXT

    PackageDependencyContextId Context { get; } //Return value is MDD_PACKAGEDEPENDENCY_CONTEXT
    String PackageFullName { get; }

    static boolean AreEquivalent(PackageDependencyContextId packageDependencyContextId1, PackageDependencyContextId packageDependencyContextId2);

    boolean AreEquivalent(PackageDependencyContext otherPackageDependencyContext);

    void Remove();
}
}

Open Questions

Q: Package dependencies are only resolved for Framework packages. Should other package types (Main, Resource, Optional) be supported?

Q: A package dependency's critieria includes user, package family, minimum version, and processor architecture. Are there other qualifiers we should be consider?

Q:WinRT: How should 'lifetimeArtifact' and MddPinPackageDependency_LifecycleHint* be expressed in the WinRT API? Some ideas:

• String file; String regkey; if both null then it's Process
• String lifetimeArtifact; boolean isFile; boolean isReg; boolean isProcess; and only 1 can be true
• ILifetimeArtifact property with multiple implementations e.g. FileLifetimeArtifact = { string file; } vs RegistryLifetimeArtifact = { HKEY root; string subkey; } vs Process=null
• ?

[jonwis]

Looks pretty reasonable. A few issues:

• Don't use declspec(dllimport/export) ... use STDAPI and use a .def file & import library for the "flat C" flavor of the API
• Use enum class Foo : uint32_t when declaring enumerations (also lets you drop the Foo_Bar prefix on everything.)
• The ...NounAreEquivalent should be ...PluralnounAreEquivalent
• PinPackageDependencyOptions needs a default ctor so it can be constructed
• Should the ...Options types be struct typed since we're being a little looser with versioning here?
• For the context type, use a "strongly typed cookie" - the pattern is struct ThingContextId { UInt64 Id; } and then there's methods like static ThingContext GetFromContextId(ThingContextId), etc. ... it's not clear the public surface needs to convert to-from a HANDLE here anyhow, can that be removed?
• Feel free to remove the static AreEquivalent method and keep a bool Equals member method instead.
• The PackageDependencyContext should probably implement IClosable - see internal Patterns document for more on this (coming to the public soon!)

[DrusTheAxe]

Thanks Jon!

Approved. Updated the proposal

• declspec/STDAPI
• Pluralnoun NOTE:It's ArePluralnounEquivalent
• PinPackageDependencyOptions ctor
• ICloseable
• context strongly-typed-cookie

Rejected:

• Equals The comparisons are for functional equivalency. PackageDependency objects are equivalent if the subset of underlying data match (e.g. same pkgfamilyname but different lifetime artifact). Equals implies same, but that would involve more data than we're checking here. The purpose is "Do these 2 definitions define the same criteria for package resolution?" Hence, 'Equivalent' not 'Equals'. 'PackageResolutionCriteriaAreEqual' is more accurate but a mouthful I'm open to suggestions Ids are strings thus trivial for callers. We could add Equals methods as { return id1.CompareNoCase(id2) } if folks feel that's useful. This is independent of equivalency methods

Open issues:

• ...Options = struct => Good question. Do our API guidelines cover this yet? Would struct restrict us to foundation data types? We need boolean, integers, strings and enums. I could see a use down the road for lists; those aren't compatible with structs, right? If we make a runtimeclass FooOptions2 { struct Foo Foo; ...new v2 properties... } where v1=struct v2=runtimeclass that inconsistency isn't a big deal (if/when that day comes)?

[marb2000]

Q: Will this work downlevel?

[DrusTheAxe]

Q: Will this work downlevel?

Yes! The target is down to whatever releases Project Reunion supports.

A design proposal is coming shortly. That'll explain 'how' we can make the magic happen. Please stay tuned

[stevenbrix]

How exactly does this work from an end user perspective? As an app developer, do I have to call these APIs in my main method? Or will there be something in the SxS manifest that declares the package dependency?

[wjk]

Question about this design. I presume the AppX packages that would be referenced using this API would be framework packages. That’s perfectly OK, until we start implementing APIs with broker processes that might need to be run in full trust. Per that linked doc, framework packages cannot declare either applications (making it awfully hard to run the broker as a separate app for boundary purposes) or capabilities (meaning we are limited to AppContainer-only APIs, since we can’t specify runFullTrust). As I understand it, only framework packages can be listed as dependencies in AppX manifests, and modification/optional packages must specify the exact PFN that they expect to be loaded into, so those won‘t work either. Could someone please clarify this? Thanks!

[DrusTheAxe]

@stevenbrix Yes, these are APIs apps would call at (un)install and runtime. The basic flow is

• Call Pin @ install-time
• Call Unpin @ uninstall-time
• Call Add in your app before using the framework's content. main() is a common answer but could be later if you choose
• Call Remove in your app when done with a framework. Or don't call at all and it's effectively done at process termination

For a rough comparison to an MSIX (packaged) application...

• Pin is akin to declaring a in appxmanifest.xml
• Add is akin to...
• ...the Deployment stack resolving that to a specific package
• ...updating the process' package graph so the specific framework's content can be found by LoadLibrary, ActivateInstance, etc
• Remove has no direct analogue in an MSIX application
• Unpin is akin to uninstalling a package for a user e.g. packageManager->RemovePackageAsync()

A packaged app has a package graph from process creation until process termination. In fact, all process have a package graph, it's just empty/null for non-packaged processes. Either way, a process' package graph is set at process creation (be it a list of packages or empty/null) and is is constant until process termination. Dynamic Dependencies supplements that static package graph with the ability to dynamically alter a process' package graph at runtime.

Unfortunately SxS is no help as Windows lacks the hooks to setup the necessary information at process creation. But let's assume for a moment we could change Windows Would that be helpful? When would you find it preferable to use the SxS manifest vs the API? Would it still be interesting if it was limited compared to the API e.g. if MddRemovePackageDependency() couldn't remove such package dependencies? Would this SxS option be attractive if it required a future version of Windows?

[DrusTheAxe]

@wjk Yes, your understanding is correct.

The Dynamic Dependency API targets Framework packages. That's both a pressing need and a natural fit for how framework packages work -- used by other packages, don't provide their own process/runtime identity etc.

Can you elaborate on what you'd like to accomplish (that frameworks don't suffice)?

[wjk]

@DrusTheAxe See #62 for an example. If that proposal were to be implemented, we would not be able to use a framework package to deploy it, as it needs to supply a full-trust helper application that uses App Services to communicate with the main app, and framework packages cannot define those. Thanks!

[DrusTheAxe]

@wjk For some of our planned features we need brokering so AppContainer/Universal apps can get access (with appropriate control) to other user content. We’ve discussed a model like this:

• Define a framework package with the in-process part of the brokered operation
• Define a main package with an appservice extension and the runFullTrust capability
• The UWP app calls into code from the framework package, which calls through the appservice entrypoint
• The full-trust component does what it needs to achieve the goal This is how an app can implement brokered access from an AppContainer/Universal app to the clipboard or other functionality. But instead of everybody building their own one-off "access the clipboard through my custom broker implementation" Project Reunion will do the work. We can provide a common implementation and apps just call "ProjectReunion.Clipboard.RequestWriteAsync(dataPackage)"

In this picture the framework and main package work collectively to provide the end-to-end integrated solution. Better together :-)

Yay for packaged applications. The DynamicDependencies proposal goes one step further, providing access for unpackaged Win32 (MSI, setup.exe, xcopy-deploy, ...) apps as well as packaged apps.

We're still mulling over whether AppService is the right technology for this, and how to ensure your Win32 application can co-deploy that Framework Package and its Main Package helpers. Stay tuned real soon for more on that front.

[DrusTheAxe]

I've posted a PR with a spec for this proposal => Spec for MSIX Dynamic Dependencies #108

Feedback is welcome 😃

[stevenbrix]

@DrusTheAxe would it be possible to use the same manifest file for unpackaged apps? This seems like the appropriate time and place to converge the appxmanifest and sxs manifest into something that works in all scenarios. Then we don't need to put an extra burden on the app author, or tooling to support multiple different techs, which I imagine will just continually diverge over time. The build can just produce a ProjectReunionManifest.xml, which is an appxmanifest.xml with some metadata that says "this app isn't running in a container". Then when the app launches, the system reads the manifest file and does the whole pinning of dependencies that you are describing.

[DrusTheAxe]

@stevenbrix > would it be possible to use the same manifest file for unpackaged apps? This seems like the appropriate time and place to converge the appxmanifest and sxs manifest into something that works in all scenarios

Are you asking/suggesting if it's possible to make the Dynamic Dependency behavior declarative and automagically happen at process creation, with the right declarative work by a developer?

Or could we merge SxS and MSIX manifests to have a single artifact?

Or could we leverage SxS to avoid the need for a helper 'sidecar' Main package?

Or something else?

Are you aware of the recent SxS support to add package identity to a non-packaged app via <msix>? Is that related to your query? Though that's more a complementary use of SxS and MSIX than a merging of manifests.

Hoping to get a clearer idea what's on your mind before trying to answer.

[stevenbrix]

Are you aware of the recent SxS support to add package identity to a non-packaged app via ? Is that related to your query? Though that's more a complementary use of SxS and MSIX than a merging of manifests.

@DrusTheAxe I was not aware that already existed, that's amazing!! That's a huge part of the puzzle, so it's cool to see that already working. Could this be extended to make https://github.com/microsoft/ProjectReunion/issues/55 a reality?

Or could we merge SxS and MSIX manifests to have a single artifact?

Or could we leverage SxS to avoid the need for a helper 'sidecar' Main package?

Yes and yes 😄

Ultimately, it would be nice to have a single manifest, so that we can have a single build pipeline, single winrt activation story, single manifest designer in visual studio, and single inner-loop workflow for developers, and single next best thing.

IMO, a perfect design wouldn't require me (as an app developer) to do anything extra, like having to pin dependencies or give myself identity in an unpackaged app. All of this should be in the manifest (which it already is for packaged apps), and the system should just take care of it for me when my app launches.

Here's some rough c++ psuedo-code to describe what I'm talking about:

bool hasManifest = ExistsReunionManifest();
if (hasManifest && !IsRegistered())
{
   // App isn't registered. Loose register, this can do 3 different things:
   // 1. Install MSIX dependencies
   // 2. Grant identity
   // 3. Run in container
   RegisterDynamicReunionManifest();
}

And yes, any one of these things can fail during startup, so there needs to be an elegant way the app can handle this. I don't show this, but I know we can figure that out. Even if it were designed in such a way that higher level frameworks like WPF or WinUI handle this special behavior, that's fine by me.

This makes our story very simple:

1. You build an app with a manifest, and the exact same build pipeline and app startup path are used for all Windows apps
2. To provide the best experience for customers, you put your app and manifest in an .msix, and submit it to the store/winget.

[DrusTheAxe]

@stevenbrix

a perfect design wouldn't require me (as an app developer) to do anything extra, like having to pin dependencies or give myself identity in an unpackaged app

Of course. ProjectReunion is striving to unify and expand the Windows platform but, much like Rome, Windows is big and broad and wasn't built in a day. It'll take us time to bring some features to where we all desire them to be. And some features are more difficult than others to make available as smooth and seamlessly as we'd like, without at least some changes to Windows itself. That's why feedback like yours is so important, to help us understand where we should focus our efforts. To identify where we can provide great solutions today, and where we need to invest in foundational improvements to improve them going forward.

Here's some rough c++ psuedo-code to describe what I'm talking about:

What if I said you could do that today?... 🤨

When we built the plumbing described in Identity, Registration and Activation of Non-packaged Win32 Apps we also identified how it plays well with the 'XCOPY-install' model -- no 'install', just dump some files on a machine, run an app and have it discover on the fly if work's needed and handle it. I don't see this mentioned in the blog post (or elsewhere); I'll follow up on that. But it's actually quite easy to do, given the new features outlined in that blog:

1. Create a 'sparse package'
2. Create your exe with an embedded SxS manifest containing
3. In your main() detect if (not running with identity) then { Install(mysparsepackage.msix); relaunch self }

Step 2 is the key here

Historically, activating a packaged app required Windows (ultimately) go through ActivateApplication() to start the process appropriately (package identity, RuntimeBehavior, TrustLevel and other manifest activation information. Let's call this Activation-via-ActivationManager.

You could make a DesktopBridge application in the classic Win32 sense (Winmain etc) instead of a Universal app, but you couldn't just run the executable. CreateProcess("foo.exe") would fail.(a)

(a) Yes we've added <Extension Category="windows.appExecutionAlias"....> that can make some of those scenarios work. A handy improvement, but not a complete solution. Not quite the same as just double-click an exe in Explorer and run.

The <msix> element is the game changer.

During process creation, if the executable has a SxS manifest containing <msix> we check if that package is registered for the user and if so, we create the process just like ActivateApplication() would! Let's call this Activation-via-CreateProcess 😄 Regardless whether you CreateProcess("kittens.exe") or ActivateApplication("Kittens_1.2.3.4_x86__1234567890abc!App") you get the same result. The same process with the same package identity and other properties.

Now here's the the critical piece.

If process creation sees <msix> in the SxS manifest but that package is not registered, we create the process anyway, as if there was no <msix> in the SxS manifest. We don't fail.

This is where step 3 comes in.

The kittens.exe process will be created whether or not the Kittens package is registered for the user. You can detect this in your app and adapt e.g.

int main()
{
    // Are we running with package identity?
    UINT32 n = 0;
    if (GetCurrentPackageFullName(&n, nullptr) == ERROR_NO_APPMODEL_PACKAGE)
    {
        // Nope. Let's correct that...
        ...install kittens package e.g. packageManager.AddPackageByUriAsync()...
        ...launch self...
        ...quit...
    }

    // We're running with package identity! Let's roll
    ...
}

Per your checklist...

1. MSIX dependencies installed? Check.
2. Identity granted? Check.
3. Run in container (and/or other activation properties, as specified in your appxmanifest)? Check.

Voila! All the runtime benefits, with no explicit, user-driven or visible 'install' step.

You can further extend this with Dynamic Dependencies per your pseudo-code's RegisterDynamicReunionManifest(). they're complementary techniques.

One caveat of this 'XCOPY-install friendly' technique is it requires Windows >= 10.0.19041.0 (aka May 2020 Update aka 20H1).

How does that sound?

[stevenbrix]

When we built the plumbing described in Identity, Registration and Activation of Non-packaged Win32 Apps we also identified how it plays well with the 'XCOPY-install' model -- no 'install', just dump some files on a machine, run an app and have it discover on the fly if work's needed and handle it. I don't see this mentioned in the blog post (or elsewhere); I'll follow up on that. But it's actually quite easy to do, given the new features outlined in that blog:

Thanks for sharing a link to this blog. The blog mentions that there is no VS support for this, are there any plans to add VS tooling for all of this to make it smoother?

The meta point I'm trying to make is a) I don't see anything in these specs about how these can be tooled so that they are easy for developers to accomplish and b) I don't want there to be separate tooling for unpackaged and packaged apps. I'm glad this all seems to be doable, but I don't understand how I can accomplish it, and it looks like way too much work to do.

[DrusTheAxe]

@stevenbrix

are there any plans to add VS tooling for all of this to make it smoother?

Are you referring to the blog post, this Dynamic Dependencies feature or something else? Just want to make sure I understand the question

a) I don't see anything in these specs about how these can be tooled so that they are easy for developers to accomplish

Agreed, it's not just a matter of it can work, it should also be easy to accomplish. Tooling's been an active area of discussion and investigation. What sort of tooling would you desire? Is something akin to VS' current new-UWP-project what you have in mind? GUI property pages etc to add/edit? Is VS your environment of interest? VSCode? Other?

b) I don't want there to be separate tooling for unpackaged and packaged apps

I expect not. After all, it's all just Windows development. We're actively trying to erase the lines between those development experiences :-)

I'm glad this all seems to be doable, but I don't understand how I can accomplish it, and it looks like way too much work to do.

Which part(s) seem too-much-work?

[stevenbrix]

are there any plans to add VS tooling for all of this to make it smoother?

Are you referring to the blog post, this Dynamic Dependencies feature or something else? Just want to make sure I understand the question

@DrusTheAxe I was referring to the blog post.

? a) I don't see anything in these specs about how these can be tooled so that they are easy for developers to accomplish

Agreed, it's not just a matter of it can work, it should also be easy to accomplish. Tooling's been an active area of discussion and investigation. What sort of tooling would you desire? Is something akin to VS' current new-UWP-project what you have in mind? GUI property pages etc to add/edit? Is VS your environment of interest? VSCode? Other?

Developers shouldn't have to write code in their main method. Maybe this is fine for frameworks where the app author owns the main method. But in WPF and WinUI, the main method is generated by tooling.

I'm glad this all seems to be doable, but I don't understand how I can accomplish it, and it looks like way too much work to do.

Which part(s) seem too-much-work?

The whole end-to-end, I don't understand what steps I need to go through in order for this to work. It would be helpful to see a doc that outlines how a customer will be successful.

Also, how does this play into the concept of .NET Core SelfContained apps? The promise there is that the app carries all of it's dependencies with it and the app author has full control. This is currently the recommended approach from the .NET team. If an app author sets SelfContained to true in their project file, will that ensure the framework package is never updated? How do we educate developers in this scenario that some of their dependencies are bundled with the app, but then some are not?

[DrusTheAxe]

We're discussing tooling and integration options to make this as friction-free as possible. The end-of-end story spans multiple parts - Dynamic Dependencies, the #156 Deployment Information API and some tooling and additional APIs to further shrinkwrap the picture.

The generated-main issue is a good one. I'll follow up on that.

Some folks are looking specifically at the .NETCore space. I've heard SelfContained come up but I don't know the latest thinking (yet :P). StayTuned.

[stevenbrix]

@marb2000 can help with SelfContained, he has done a lot of work in this space already.

[DrusTheAxe]

The generated-main issue is a good one. I'll follow up on that.

XAML ask => Proposal: Extensibility hooks in generated main #3408

WPF ask => Proposal: Extensibility hooks in generated main #3632

[wjk]

@DrusTheAxe #158 seems to have been lost in the transition from master to main. It shows as merged on GitHub, but is not present in the main branch (and the master branch it was merged into has been deleted). Since the source branch has since been deleted also, it is now impossible to use the MSIX DD code, as it now exists only in the form of diffs on the PR page; it cannot be checked out now. Could someone please resuscitate this code and get it into main? Thanks!

[DrusTheAxe]

@wjk Yes something went weird with the branching. I've killed the PR. Rebase'ing against main shows some merge conflicts :-(

Let me get the branch back up and verified it's current.

[DrusTheAxe]

OK I made a new branch user/drustheaxe/dyndep that's rebased against the latest in main. All code preserved and builds and tests pass so yay.

[DrusTheAxe]

Feedback from recent design reviews centered on the 'helper' main package and asks to do something simpler / more streamlined for developers. So we made a change to plan - Project Reunion will provide the 'helper' main package for folks using Dynamic Dependencies. Thus is born the 'Dynamic Dependencies Lifetime Manager'.

The branch has the core implementation. I'm working on an update to the spec about this, plus some remaining work e.g. integrate the DDLM into the CI/Build pipeline. Coming soon

[DrusTheAxe]

Mission accomplished, just forgot to resolve the issue