search

KhronosGroup / MoltenVK

MoltenVK is a Vulkan Portability implementation. It layers a subset of the high-performance, industry-standard Vulkan graphics and compute API over Apple's Metal graphics framework, enabling Vulkan applications to run on macOS, iOS and tvOS.
Apache License 2.0
4.76k stars 419 forks source link

support for vulkan video decode/encode apis #1337

Open maddanio opened 3 years ago

maddanio commented 3 years ago

Hello

will MoltenVK implement the new vulkan video encode/decode apis?

https://www.khronos.org/blog/an-introduction-to-vulkan-video

best

cdavis5e commented 3 years ago

While I'm pretty sure Metal already has support for hardware-accelerated video encoding/decoding, until Apple exposes it publicly, there isn't much I think we can do until then.

cdavis5e commented 3 years ago

Actually... we might be able to use the VideoToolbox framework to support this.

maddanio commented 3 years ago

yes, that was the idea. we are already doing vulkan<->vt exchange with metal surfaces. if it helps I can isolate the neccessary steps that are proven to work for us.

maddanio commented 3 years ago

it's not particularly urgent for us, especially as long as support for this is still sparse elsewhere (intel, amd), but it came up and I felt it should be a relatively mechanical task to implement and nice to support the initiative by khronos

mbechard commented 3 years ago

I'll mention that we'd find this super useful. Being able to use hardware encode/decode via a single API across multiple platforms will be a huge time saver.

oscarbg commented 2 years ago

+1

ThomasHagenJohansen commented 1 year ago

+1

lu-zero commented 8 months ago

yes, that was the idea. we are already doing vulkan<->vt exchange with metal surfaces. if it helps I can isolate the necessary steps that are proven to work for us.

Which are the steps?

maddanio commented 8 months ago

so, let's see, been a while :). so for the decode side

  1. create a CVMetalTextureCache for the device
  2. given a CVImageBufferRef (which you get from core video) create one metal texture per plane using CVMetalTextureCacheCreateTextureFromImage
  3. create a VKImage for each plane that is compatible with the right properties
  4. use vkSetMTLTextureMVK to set the image to the texture obtained via CVMetalTextureGetTexture

for the encoding side its basically the same, only then you render to the obtained texture on the vulkan side and use the pixel puffer on the core video side

i hope this helps. I can also try to isolate some code, but that may take longer, as its tangled up in our codebase a bit

maddanio commented 8 months ago

ah and currenlty this means you have to link directly to moltenvk and not use any layers, unless they made it possible now to "see through" layers as I think was requested a while ago