search

DiligentGraphics / DiligentCore

A modern cross-platform low-level graphics API
http://diligentgraphics.com/diligent-engine/
Apache License 2.0
622 stars 137 forks source link

Zero copy hardware video decode and encode. #75

Closed mateli closed 3 years ago

mateli commented 5 years ago

There are some samples on how to play video with a combination of OpenMAX, EGL and OpenGL ES on Raspberry pi. For example there is code to play video on the surface on rotating 3D objects. I have failed to find something similar on other platforms but I know that firefox, chrome and gstreamer can do something similar with OpenGL and/or VAAPI. The question here is if we can take such code and make it platform independent. I have been unable to find information on how to make OpenMAX or VAAPI render directly to Vulcan.

Another interesting thing that I would like to see is rendering 3D directly to a video stream which I know that nvidia has very platform dependent API:s for. This can be used for streaming, recording or running applications remotely.

These could be two different issues but in a "traditional" solution OpenMAX, EGL and OpenGL (ES) would probably be the solution. However OpenMAX is marked as deprecated and EGL isn't made to work with Vulcan.

My impression is that game developers usually do software decoding and render video frame by frame on a surface. This is really not optimal especially for low end CPU:s. I have several AMD APU:s where playing H.265 in software isn't an option but where VAAPI runs very smoothly. And on Raspberry PI playing pretty much anything in software isn't going to work.

There are also some that use VAAPI or OpenMAX to decode and pass the return data to OpenGL. However this may actually be slower due to the two extra memory copies. What is fast is to upload the encoded frame to the VPU and have it write it directly to a GPU buffer that is then rendered on the surface that it is mapped to.

DiligentGraphics commented 5 years ago

For something that is not cross-platform or is API-specific, Diligent offers a couple of options. First, API-specific features can be exposed through API-specific interface such as IDeviceContexD3D11. Second, there is native API interoperability that provides access to internal objects, states etc. It allows building extensions that are not part of the core API. One such extension is asyncrhounous texture uploader (https://github.com/DiligentGraphics/DiligentCore/blob/master/Graphics/GraphicsTools/include/TextureUploader.h). It looks like what you are talking about is very platform and API-dependent, so native API interoperability seems to be the better way to go here

mateli commented 5 years ago

Well the core problem is that I don't really know how to do it outside of OpenGL ES where there are implementations for Raspberry Pi. What I really want is to make games and applications that can mix 3D and video without bothering with low level API:s. But the problem is that the "standard" way of playing video on a Vulkan surface is trough software decoding. Vulkan has an API to draw on a texture from software that I guess that you are using in TextureUploader.

However I do not want to upload a texture I want the VPU to draw directly on the texture. This is how I know to do it.

  1. Upload encoded frame to GPU/VPU memory
  2. Decode using OpenMAX/VAAPI
  3. Download decoded frame to system memory
  4. Reupload decoded frame to Vulkan surfface
  5. Render with Vulkan

What I want to do is this:

  1. Upload encoded frame to GPU/VPU
  2. Decode using OpenMAX/VAAPI/Whatever
  3. Render with Vulkan

What I can do but only tried on Raspberry pi:

  1. Get a buffer from EGL
  2. Upload encoded frame with EGL
  3. Decode with OpenMAX
  4. Render EGL buffer with OpenGL ES.

The last solution should work on any platform that supports OpenMAX. But it uses EGL for buffer sharing which as far as I know is not possible with Vulkan. From the name "TextureUploader" I don't think that's what I'm looking for...

And yes the solution for this is probably different on different platforms. That is why I both need to figure out how to do this on several platforms and have an abstraction for it because the applications that use it should not have to bother with that.

DiligentGraphics commented 5 years ago

I mentioned texture uploader as an example of cross-platform component that is not part of the Core API. The video decoder can be implemented in a similar way. I haven't played with hw-accelerated video decoding but I'm pretty sure there are ways to share decoded images with rendering in all graphics APIs, especially since in many cases the decoder resides on the GPU. So there should be a cross-platform way to do this