Support hardware encoding for CPUTexture

[SebastianKunz]

As of now there is only hardware encoding support for DirectX11 2D Texture. I would highly appreciate if you expand the hardware encoder for more formats. Mainly plain pixel data (RGBA, BGRA, ...). In case this is not planned, could you give some pointers on what to do to implement such an encoder? I looked trough the webrtc-native-nvenc project, but struggle with finding a way to send raw pixel data to the GPU.

Cheers, Sebastian

[ziriax]

So you mean plain pixel data living in the CPU RAM?

Older versions of NVENC only supported the BGRA32 format (and NV12), but newer hardware and SDK seems to support RGBA32 too... See NV_ENC_BUFFER_FORMAT in nvEncodeAPI.h

I don't know know what cards support this though.

NVidia's nvpipe uses CUDA to convert the pixel formats.

But in our case, the Direct3D renderer is running on a different thread, and we couldn't find a way to have both Direct3D and CUDA on different threads, so we removed the nvpipe library, and directly used the NvEnc D3D11 encoder classes, only using Direct3D API calls.

We used to have support for VideoFrameFormat.CpuTexture (a pointer to plain pixel data in memory, in BGRA32), but that was removed in the above patch. CpuTexture is actually a very bad name, and should be removed... Since you configure what codec to use globally, the H264 encoder should also support the VideoFrameFormat.RGBA32 etc. And that is exactly your request 😄

Now, if you have plain pixel data on the CPU, it wouldn't be too difficult to copy this to a Direct3D11 texture: docs gist

If the pixel format is not supported by NVENC, you would have to convert it, either on the CPU or the GPU. That is harder, but libraries most likely exist for this. With the GPU, you would have to either write a compute shader, or a full-quad fragment shader, and make sure to do the right thread locking.

So in summary, to support the other pixels formats, a lot of work needs to be done. At least the following files should be modified:

• PeerConnection.cpp: each video track should remember what codec is used (we currently use the same codec for all peer connections). The CpuTexture format should be removed, and depending on the codec type of the video track, either send a webrtc::NativeVideoBuffer or webrtc::I420Buffer.

• NvEncoderH264.cpp: this should support the other pixel formats, and methods should be added to the encoder to copy the CPU pixels to the GPU textures already created for encoding.

[SebastianKunz]

Yes exactly! I am now capturing the frame using webrtc's built in desktop_capture module (using directx). Those frames are copied to RAM and I then want to send them to the gpu. And I managed to get it to work. I used the NvEncoderH264 class as a reference and basically rewrote the encoding part using NvPipe (it was actually, not that difficult). However this is stupidly inefficient, copying about 2 GBs of pixel data per second from cpu to gpu (4k 60fps). Luckily webrtc supports a SharedMemoryFactory that allows to copy the frame to gpu memory. I will try it out and see how it goes.

Oh and there is one more problem. The R and B channels of the frame are swapped. Both nvpipe and the capture device specify that they use RGBA pixel format. So one of them is lying. Do you know anything about this? My gpu is up to date, so it should support RGBA.

[ziriax]

Yes, it is really silly to copy GPU frames to the CPU RAM and back 😉

RGBA means different things depending on what API is used, I blame little endian CPUs for that confusion 😉

I would try to pass NV_ENC_BUFFER_FORMAT_ARGB and NV_ENC_BUFFER_FORMAT_ABGR to the NVENC encoder, and see what works.