Question: is there a way to plug a D2D1 pixel shader into a composition pipeline?

[Sergio0694]

Background

I'm working on adding D2D1 support to my library ComputeSharp, which allows developers to write HLSL shaders entirely in C#, and then takes care of all the work to transpile them to HLSL, compile them, and offer additional APIs to eg. run them as compute shaders over DX12 APIs. Recently I've started working on a new ComputeSharp.D2D1Interop library (see here), with the goal being of allowing Paint.NET to use this to replace all the current infrastructure with separate HLSL files for the pixel shaders, a pre-build MSBuild step to compile them, and then additional logic to load them and extract the constant buffer for dispatch. The new APIs in ComputeSharp instead allow developers to do everything in C#, and very easily. For instance:

[D2DInputCount(2)]
[D2DInputSimple(0)]
[D2DInputSimple(1)]
[D2DEmbeddedBytecode(D2D1ShaderProfile.PixelShader50)]
[AutoConstructor]
public readonly partial struct MyBlendShader : ID2D1PixelShader
{
    public float factor;

    public float4 Execute()
    {
        float4 pixel0 = D2D1.GetInput(0);
        float4 pixel1 = D2D1.GetInput(1);

        return Hlsl.Lerp(pixel0, pixel1, factor);
    }
}

This is then transpiled to an HLSL pixel shader automatically at build-time, and embedded into the assembly.

Authors can then use it like this:

MyBlendShader shader = new(0.6f);

// Get the shader bytecode
ReadOnlyMemory<byte> bytecode = D2D1InteropServices.LoadShaderBytecode<MyBlendShader>();

// Get the constant buffer with all shader properties
ReadOnlyMemory<byte> buffer = D2D1InteropServices.GetPixelShaderConstantBufferForD2D1DrawInfo(in shader);

This provides a major productivity improvement for developers:

• Everything is done in C#, no need for C++ APIs
• No HLSL knowledge is required
• No additional tooling required, no build steps
• Shader code gets IntelliSense support (!)

Additionally, I also have more lowlevel APIs to easily register a pixel shader effect manually from an ID2D1Factory1:

// Register a pixel shader effect for a target factory
D2D1InteropServices.RegisterPixelShaderEffectForD2D1Factory1<MyBlendShader>(d2D1Factory, out Guid effectId);

This effect can then be created normally from a factory using the returned id, to get an ID2D1Effect object. This effect will also automatically handle insertion of the node into the transform graph. Users also have the ability to define a custom draw transform mapping, through additional APIs not shown here.

Now, obviously something that came to mind is: it would be awesome if we could leverage this to also empower developers to easily write pixel shaders to insert into a Win2D/Composition pipeline, to realize all sorts of custom effects. Unfortunately though, I can't seem to find a way to do this, as the documentation says that the PixelShaderEffect type doesn't support composition (it's marked with [NoComposition]). I know it would still be possible to draw stuff with a pixel shader either through a swap chain panel (which I also support, via DX12 APIs), or with a canvas drawing session, but neither of those has the same usability that just having an effect to plug into a composition pipeline provides.

Open questions

I'd like to explore this to ideally augment our APIs in the Microsoft.Toolkit.Uwp.UI.Media package (see docs here) (say, in some hypothetical Microsoft.Toolkit.Uwp.UI.Media.PixelShaders package), but I'm not sure how or if this is actually supported (or, if it is, I don't see this being mentioned in the docs, along with an explanation of how to set this up).

Essentially:

• Why is PixelShaderEffect marked as [NoComposition]?
• Is there really no way at all to use this effect in a composition pipeline?
• Does the same limitation apply to both UWP and WinUI 3?
• Is there some other way to set this up, even if through additional manual work to set things up? This includes either through PixelShaderEffect, or through some more lowlevel building blocks, such as eg. having to manually query for a D2D1 factory, register the effect, create it, and then insert it into a graph somehow, as long as it's documented.

I really think this could potentially give developers an incredibly powerful set of new tools to interact with the visual layer.

Pinging @codendone and @marb2000 as a follow up to our past conversations on composition APIs. Feel free to cc. others if you know someone that works in this area in particular that you think would be able to shed some light on this. Thanks! 😄

[sylveon]

Pixel shaders are blocked in composition pipelines because composition is implemented in a privileged process (DWM) and to use the pixel shaders DWM would have to load them. This is a security issue.

Another reason is that this would allow readback of the screen contents behind the app window by the pixel shader, which is a privacy issue.

[Sergio0694]

Right, I'm aware of that, but I'm wondering if there still isn't some way to achieve that. Or, if the issue is allowing readback on the screen for privacy reasons, then would that mean that'd be fine in WinUI 3 apps, since those are fulltrust and can already do whatever anyway? My point is - being able to plug a pixel shader into a composition pipeline in some way would enable a ton of incredibly cool scenarios for developers, and I think this scenario is something that would be worth exploring 🙂

"Another reason is that this would allow readback of the screen contents behind the app window by the pixel shader, which is a privacy issue."

I'm also confused about this statement - if a pixel shader is just inserted into the effect graph by the composition pipeline, wouldn't it mean it could just read from its inputs and write to the output, which is then just passed along into the composition pipeline? As in, it would not allow for eg. saving that backbuffer directly somewhere else. Not to mention, the background you get from a host backdrop is already pre-blurred anyway for privacy reasons, so I don't see an issue with accessing that 🤔

[pjmlp]

So yet another thing that WPF does better than WinUI, I wonder if the team actually knows what they are trying to replace.

• ShaderEffect Class

• PixelShader

[Sergio0694]

@pjmlp Please let's avoid off-topic comments, that doesn't really help nor contribute to the conversation 🙂

Just really looking forward for someone in the comp/rendering team to chime in, so we can figure out a way to do this 😄

[Sergio0694]

Bumping this (see https://github.com/microsoft/microsoft-ui-xaml/discussions/8638).

[duncanmacmichael]

Bumping this issue - pinging @codendone and @ChewWorker to see if we can answer any of Sergio's questions.

[Sergio0694]

A small update on this. I figured out a way to do this (at least for several scenarios):

• Create a Win2D CanvasSwapChain
• Use CanvasComposition.CreateCompositionSurfaceForSwapChain to get an ICompositionSurface bound to it
• Plug that surface into your Composition tree (eg. via a surface brush wired up to a sprite visual)

[!NOTE] This only helps in the shader -> composition case. Doesn't help if you want a shader working on composition sources.

Now, I have good news and bad news.

• Good news: this works
• Bad news: this only works on UWP

Possibly related to #5025 as part of a larger effort to bring back global composition on WinUI 3 too? 🙂