d3d11 backend for the Render API

[leavittx]

It would be very nice to have d3d11 interface accessible within libmpv/render.h for external applications.

It could be a solution which allows to get directx 2D texture/shader resource view handles which can be used in a user application with its own dx11 device/context (i.e. shared immutable or dynamic texture). The final video frames are to be rendered into that texture (app tells needed resolution to mpv)

The alternative solution is to pass a d3d11 device from host app to mpv and then either (a) again, create a texture inside mpv and pass back to an app or (b) do it similar to how GL render API works now, i.e. render to a user-provided render target (which I don't like that much though, since having a texture is more convenient to me, though it still serves its needs)

I started do implement this feature on my own some time ago, but I'm a bit lost inside mpv's internal rendering structures, and debugging is quite tricky under windows (although I have a pdb file which I can use for debugging under Visual Studio, but it's not ideal).

Just wondering if someone can guide me a bit?

[leavittx]

@rossy Maybe you have some thoughts? I wrote you an e-mail couple of weeks ago, not sure if you received it

[rossy]

@rossy Maybe you have some thoughts? I wrote you an e-mail couple of weeks ago, not sure if you received it

Sorry about that. I received your email and I thought I'd spend some time to think about what a d3d11 interface would look like so I could write a proper response, but it slipped my mind.

It could be a solution which allows to get directx 2D texture/shader resource view handles which can be used in a user application with its own dx11 device/context (i.e. shared immutable or dynamic texture). The final video frames are to be rendered into that texture (app tells needed resolution to mpv)

Using two devices and texture sharing is possible, and it would mean mpv wouldn't have to worry about saving/restoring the device context state, but I think this comes with extra synchronisation overhead, since it seems like D3D11 keyed mutexes block on the CPU while they wait for the texture to become available. Avoiding pipeline stalls with multiple devices would probably require a queue of surfaces. Because of this, I think using a single shared device would be easier and more efficient.

The textures can't be immutable or dynamic either. In D3D11, texture usage flags govern how textures can be accessed by the CPU. Immutable and dynamic textures can be written either once or multiple times by the CPU, and can only be read by the GPU. Since video frames will be rendered on the GPU, the textures will have to have default usage (D3D11_USAGE_DEFAULT.)

The alternative solution is to pass a d3d11 device from host app to mpv and then either (a) again, create a texture inside mpv and pass back to an app or (b) do it similar to how GL render API works now, i.e. render to a user-provided render target (which I don't like that much though, since having a texture is more convenient to me, though it still serves its needs)

Yeah, I think passing a D3D11 device from the API user to mpv is the way to go, at least for the first version of this API. If an API user really needs device sharing (for example, to share textures across process boundaries) they could still do it, they'd just have to create both devices. Since mpv is pretty flexible with what kinds of devices it works on (with feature level, WARP, etc.) that shouldn't be a problem.

I'm not sure why (b) would be less convenient for you. To be clear, you can still use textures. The only difference is that the CreateTexture2D call would be in your code rather than mpv's. I'm not sure if mpv's API should accept the ID3D11Texture2D or the ID3D11RenderTargetView, but creating an RTV from a texture is pretty trivial (see mpv's code for an example.)

[Dixeran]

Sorry to disturb, but I would like to ask if there are any recent developments or plans on this issue?

I managed to render video into my SwapChainPanel in an UWP application by using vo=gpu and gpu-context=d3d11. It worked, but the default beavior of d3d11 context is to create a win32 window and use methods from w32_common.h, it is painful to bypass all these calls. It would be really convience if we can decouple the d3d11 context from w32, and make it as an option.

Or maybe there is a better way to achieve this goal? After all, I'm not familiar with the architecture of libmpv backend, so I directly modified some codes of d3d11 context (in a very intrusive form).

[jeeb]

@Dixeran I think you might have the best idea of what sort of embedding an API user might want. Neither me or rossy are actively working on an application requiring this, and thus we are not likely to know what would work and what wouldn't.

See libmpv/render{,_gl}.h for how OpenGL embedding currently looks like.

[ghost]

The libmpv API should be able to support this relatively easily. It worked for GL, after all, and GL has even worse characteristics for embedding than D3D11 (shared state, thread-affinity of the context). There are two layers below libmpv's "actual" render API: the render backend (currently only render_backend_gpu in libmpv_gpu.c, which reuses the vo_gpu renderer), and within that, the GL glue code (libmpv_gpu_context_gl in libmpv_gl.c). For using the vo_gpu renderer with the libmpv render API, you'd essentially have to add a libmpv_gpu_context_d3d11.

I'm not sure which kind of embedding would be best for this. Possibilities include:

• full shared state like with the render API GL backend (would require what I described above)
• some sort of shared object like a swapchain or a texture (might still use the render API)
• embedding a full window (I bet that's somehow possible even with UWP, but might require some sort of UWP support in mpv separate to w32_common.c; would probably not use the render API)

[Dixeran]

Thanks, now I have a basic knowledge about libmpv, I'll try to implement a libmpv_gpu_context_d3d11. Currently, I embed the mpv by following steps:

1. Create my custom D3D11 device and share it to mpv.
2. Let mpv create it's swapchain from that shared device, by calling CreateSwapChainForComposition(), then return a pointer to me.
3. Bind the swapchain to my SwapChainPanel.

[ghost]

If your "old" code didn't need the application to drive the render loop, it might be preferable. Is the code available anywhere?

[Dixeran]

This is my modified mpv : https://github.com/Dixeran/mpv/tree/preview, you can checkout the commits I have made. There are mainly the following changes:

• Modified: core.h
• New files: d3d11/custom_helper.h and custom_helper.c : some methods to bind custom device
• Modified: d3d11/context.c : avoid calling w32 methods.
• Modified: d3d11_helpers.c : create swapchain

And here is a basic example of UWP : https://github.com/Dixeran/mpv-uwp

[ghost]

That's interesting. It's almost like embedding with --wid (where you pass a window handle into which mpv embeds its own window), but conceptually more modern and cleaner.

Would it make sense if you simply passed a ISwapChainPanelNative to mpv? (I don't know if mpv could still get resize events or whatever).

It almost sounds like for this purpose, this method of embedding would be preferable or simpler. But if you feel like it, you can pursue the mpv_render API approach too. (It has the glaring disadvantage that the application needs to manage the swapchain and driving the render loop itself, but on the other hand it'd allow doing things like rendering into textures.)

Now to how this should be implemented so it's acceptable for merging:

• maybe you could split the d3d context code (ra_ctx_d3d11) into one that uses w32_common.h, and a different one that uses a custom swap chain (that's almost how you did it)
• this shouldn't touch the client API core stuff (should be unnecessary)
• passing the swapchain parameters could done in one of the following ways:
  • with --wid; casts a pointer to an integer to funnel it through the option API. ra_ctx_android is a newer backend that uses this method, and it needs some hacks to make it work at all. This is quite unclean, but at least doesn't require thinking too hard about it.
  • add new dedicated API (like you did it). This should be extensible and not be d3d-specific.
  • repurpose parts of the mpv_render API. This would use the render API for passing parameters and possibly invoking commands like resizing the swapchain. It would not add a render backend (like a "proper" mpv_render API implementation would do it). This would never use --vo=libmpv, but instead would use --vo=gpu, and the d3d context would "connect" the render API instead. I have to think about this.

[ghost]

Also, @rossy's input on this would be interesting, but I think he's away for this week.

[rossy]

That looks neat. Agreed that we should add some sort of official API for libmpv to render to a SwapChainPanel.

Would it make sense if you simply passed a ISwapChainPanelNative to mpv?

That should work, though a SwapChainPanel isn't the only thing that can embed a swapchain. Swapchains can also be embedded in IDCompositionVisual, and it seems like UWP also has ISwapChainBackgroundPanelNative, so maybe an API that just takes a ISwapChainPanelNative wouldn't be generic enough.

If we want it to work by the API user passing an object to mpv, instead of the API user requesting the swapchain from mpv, we could also consider requiring the API user to create the IDXGISwapChain, and having mpv render to that.

(I don't know if mpv could still get resize events or whatever).

I think technically it could, though it would require calling UWP APIs, and calling those from C is a bit of a nightmare.

[Dixeran]

I rewrote the implementation with a new idea (https://github.com/Dixeran/mpv/tree/d3d11_headless), and I found it maybe unnecessary to pass in a custom device. So I folk a new "headless" gpu context, most of the previous code was reused and only parts that init, resize and invoke the vo_w32 was modified.

I tried to make this headless context work like this: Use --gpu-context=d3d11_headless to switch to the headless one. Use libmpv API to set the swapchain buffer size and set a pointer to send out the created swapchain. When headless gpu context initializing or rendering, it will retrieve new window size from libmpv render_backend_d3d11_headless->priv. So we don't have to drive the render loop since we are still using vo_gpu, but we do need to call mpv_render_context_set_parameter when display size changed. No new API exposed.

[ghost]

Nice. Yeah, I think that's workable, the idea is good, and conceptually this can be used as is. Now, to upstream this, the details have to be cleaned up and polished. Here's a number of things that catch my eye:

• The swapchain code shouldn't be duplicated, instead a way needs to be found to make it independent of w32_common (maybe @rossy has ideas here).
• Instead of giving more code access to the internals of mpv_render_context/mpv_handle/etc., there should be a simple accessor or accessors which return whatever they need.
• Needs a new parameter instead of "redefining" MPV_RENDER_PARAM_OPENGL_INIT_PARAMS.
• The new parameters should be documented (add a comment section somewhere in libmpv/render.h).
• Since the new libmpv render backend has no actual render callbacks etc. set, most libmpv render API functions probably crash when calling them if the d3d backend is active, which shouldn't happen.
• Instead of D3D11_HEADLESS I'd probably call it DXGI_SWAPCHAIN or so? The libmpv API doesn't have anything to do with D3D11 anymore. The backend can/should still have d3d11 in the name.
• When everything is done, we need an example program (preferably desktop win32) for testing, so we don't accidentally break it when we make changes to anything related to libmpv or d3d later.

Feel free to ask any questions. Feel free to request that I do certain parts if you're not sure how to do them, but the problem is that I won't be able to mess with win32 or d3d related things too deeply. (I'm on Linux usually.)

So I folk a new "headless" gpu context,

I don't think "folk" is a verb in English, can you reword?

[Dixeran]

I wanted to type "fork" actually 😓. I'm glad it works.

• The swapchain code shouldn't be duplicated, instead a way needs to be found to make it independent of w32_common (maybe @rossy has ideas here).
• Instead of giving more code access to the internals of mpv_render_context/mpv_handle/etc., there should be a simple accessor or accessors which return whatever they need.

Regarding these two points, I don't quite understand how to modify them to meet the requirements. Can you give me some suggestions?

• Needs a new parameter instead of "redefining" MPV_RENDER_PARAM_OPENGL_INIT_PARAMS.
• The new parameters should be documented (add a comment section somewhere in libmpv/render.h).
• Since the new libmpv render backend has no actual render callbacks etc. set, most libmpv render API functions probably crash when calling them if the d3d backend is active, which shouldn't happen.

Yes, this part of code is used for testing, and I will modify this part.

-Instead of D3D11_HEADLESS I'd probably call it DXGI_SWAPCHAIN or so? The libmpv API doesn't have anything to do with D3D11 anymore. The backend can/should still have d3d11 in the name.

I agree with that. The name I wrote in libmpv is a bit confusing.

• When everything is done, we need an example program (preferably desktop win32) for testing, so we don't accidentally break it when we make changes to anything related to libmpv or d3d later.

I tend to give a desktop UWP program (can be installed by side loaded) instead of a traditional win32 program. It seems hard in win32 program to just embed a swapchain, the only way I can see is IDCompositionVisual which rossy mentioned, but it is deprecated in MSDN. Also, in my opinion, the traditional win32 program uses --wid rather appropriately.

[ghost]

Regarding these two points, I don't quite understand how to modify them to meet the requirements. Can you give me some suggestions?

For the first point, I think most code can be shared? I didn't look at your code again just now, but I think it was duplicated. We try to avoid duplicated code.

The second point: the definitions for mpv_handle and so on should remain "private" if possible. Instead, define a function that your d3d code can call, and that returns what you need from these context structs.

If you still don't know how to do these, I could take a closer look.

I tend to give a desktop UWP program (can be installed by side loaded) instead of a traditional win32 program. It seems hard in win32 program to just embed a swapchain, the only way I can see is IDCompositionVisual which rossy mentioned, but it is deprecated in MSDN. Also, in my opinion, the traditional win32 program uses --wid rather appropriately.

An UWP program is fine if traditional win32 absolutely doesn't work. Still sounds like it'd make testing rather hard.

[Dixeran]

Okay, I understand what I should do now.

[rossy]

The swapchain code shouldn't be duplicated, instead a way needs to be found to make it independent of w32_common (maybe @rossy has ideas here).

Agreed. Those changes seem to duplicate a lot of code. Some ideas to reduce code duplication:

The "d3d11" and "d3d11_headless" contexts probably don't have to be defined in separate files. Most of the functions in those files are the same, or mostly the same except the headless version doesn't call vow32*. If both the "d3d11" and "d3d11_headless" contexts were defined in context.c, they could share code (just add the second struct ra_ctx_fns to the end of the file.)

You could get different behaviour from each context by adding a headless flag to struct priv and giving the d3d11_headless context a different init function that sets priv->headless to true and calls the d3d11 init function. Then, the vow32 calls could be guarded by !priv->headless. (One problem with that would be that it's harder to remove w32_common.c from the build, but it might be best to replace the vow32 functions with stubs for a UWP-only build.)

As for mp_d3d11_create_swapchain_headless, I think you could reuse the existing mp_d3d11_create_swapchain instead of creating a new function. Just call IDXGIFactory2_CreateSwapChainForComposition instead of IDXGIFactory2_CreateSwapChainForHwnd in create_swapchain_1_2 when opts->window is NULL. Also, skip the call to IDXGIFactory_MakeWindowAssociation.

It seems hard in win32 program to just embed a swapchain, the only way I can see is IDCompositionVisual which rossy mentioned, but it is deprecated in MSDN.

Are you sure it's deprecated? The DirectComposition docs recommend using Windows.UI.Composition APIs instead for apps on Windows 10, but I don't think they've explicitly deprecated it yet.

There seems to be a way to use Windows.UI.Composition from Win32 with ICompositorDesktopInterop::CreateDesktopWindowTarget, but it would still require calling UWP APIs, which is not easy to do from mingw-w64, since most APIs aren't available.

[Dixeran]

Thank you for your very practical advice! And you are right, DirectComposition is not deprecated but not recommended, I'll see if I can use it.

[ZhenFTW]

I know I came too late, but I really hope someone made a d3d backend for libmpv. Libmpv_gpu_context_d3d11 api to pass context would be great idea.

[ZhenFTW]

I rewrote the implementation with a new idea (https://github.com/Dixeran/mpv/tree/d3d11_headless), and I found it maybe unnecessary to pass in a custom device. So I folk a new "headless" gpu context, most of the previous code was reused and only parts that init, resize and invoke the vo_w32 was modified.

I tried to make this headless context work like this: Use --gpu-context=d3d11_headless to switch to the headless one. Use libmpv API to set the swapchain buffer size and set a pointer to send out the created swapchain. When headless gpu context initializing or rendering, it will retrieve new window size from libmpv render_backend_d3d11_headless->priv. So we don't have to drive the render loop since we are still using vo_gpu, but we do need to call mpv_render_context_set_parameter when display size changed. No new API exposed.

Hello, I visited the repository and tried to understand what your new commits means but I'm so confused why the word headless??

[Dixeran]

I rewrote the implementation with a new idea (https://github.com/Dixeran/mpv/tree/d3d11_headless), and I found it maybe unnecessary to pass in a custom device. So I folk a new "headless" gpu context, most of the previous code was reused and only parts that init, resize and invoke the vo_w32 was modified. I tried to make this headless context work like this: Use --gpu-context=d3d11_headless to switch to the headless one. Use libmpv API to set the swapchain buffer size and set a pointer to send out the created swapchain. When headless gpu context initializing or rendering, it will retrieve new window size from libmpv render_backend_d3d11_headless->priv. So we don't have to drive the render loop since we are still using vo_gpu, but we do need to call mpv_render_context_set_parameter when display size changed. No new API exposed.

Hello, I visited the repository and tried to understand what your new commits means but I'm so confused why the word headless??

It means that there is no frontend created by mpv.

[ZhenFTW]

So it can't be use to display video? only api to width and height?

[jindal1979]

@Dixeran Do you plan to merge your changes into mpv. What is the plan to support Swapchain as rendering option on Winui 3 ?

[Dixeran]

@Dixeran Do you plan to merge your changes into mpv. What is the plan to support Swapchain as rendering option on Winui 3 ?

I'm not currently working on this feature, so there is no plan for merging, but you can checkout the most recent update here (https://github.com/Dixeran/mpv-swapchain). I've tested it with WinUI3 C++/WinRT and the swapchain panel widget, it works fine without the airspace issue.

[sudongg]

@Dixeran Do you plan to merge your changes into mpv. What is the plan to support Swapchain as rendering option on Winui 3 ?

I'm not currently working on this feature, so there is no plan for merging, but you can checkout the most recent update here (https://github.com/Dixeran/mpv-swapchain). I've tested it with WinUI3 C++/WinRT and the swapchain panel widget, it works fine without the airspace issue.

Can you provide any winui3 samples?