Nvidia NVENC with non NV12 (or P010LE) pixel format corrupted video

[bmegli]

May lead to corrupted data (here for BGR0 input):

[bmegli]

Quick Workaround

This is caused by changes in #26

It is enough to replace:

• https://github.com/bmegli/hardware-video-encoder/blob/c34af9aedec22d3405b0152273ff8ede541526ea/hve.c#L227-L233

With:

frames_ctx->sw_format = h->sw_pix_fmt;

As implemented before #26

[bmegli]

But we need a way to solve it for both VAAPI and NVENC at the same time

[bmegli]

But we need a way to solve it for both VAAPI and NVENC at the same time

Encoders

NVENC

ffmpeg -h encoder=h264_nvenc

Encoder h264_nvenc [NVIDIA NVENC H.264 encoder]:
    General capabilities: delay hardware 
    Threading capabilities: none
    Supported pixel formats: yuv420p nv12 p010le yuv444p p016le yuv444p16le bgr0 rgb0 cuda
# other options follow

Good - it can directly work from multiple pixel formats

VAAPI

ffmpeg -h encoder=h264_nvenc

Encoder h264_vaapi [H.264/AVC (VAAPI)]:
    General capabilities: delay 
    Threading capabilities: none
    Supported pixel formats: vaapi_vld

From #26 we know that:

• in 3.x some format conversions happened "auto-magically"
• in 4.x the same fail
  • on Intel it is possible to set hw frames sw format AV_PIX_FMT_NV12 or PO10LE and conversions will still happen
  • on AMD I have seen the same scenario fail

Full support might require:

• explicit hw upload
• with color conversion
• but we would still need to specify to what we should convert

At least Intel will work to some extent with logic like in #26

[bmegli]

For now we will:

• use frames_ctx->sw_format = h->sw_pix_fmt by default

Make special case for VAAPI to fallback to NV12 or P010LE depending on depth

I am not terribly happy but it will make Nvidia and Intel acceleration work in most scenarios

We will need to revisit it (VAAPI path):

• when generalizing non NV12 (or P010LE) input between Intel and AMD
• if using 4:2:2 or 4:4:4 chroma subsampling on modern hardware
• if using bit depth beyond 10 bits on modern hardware

    frames_ctx->sw_format = h->sw_pix_fmt;

    // Starting from FFmpeg 4.1, avcodec will not fall back to NV12 automatically
    // when using non 4:2:0 software pixel format not supported by codec with VAAPI.
    // Here, instead of using h->sw_pix_fmt we always fall to P010LE for 10 bit
    // input and NV12 otherwise which may possibly lead to some loss of information
    // on modern hardware supporting 4:2:2 and 4:4:4 chroma subsampling
    // (e.g. HEVC with >= IceLake)
    // See:
    // https://github.com/bmegli/hardware-video-encoder/issues/26
    if(frames_ctx->format == AV_PIX_FMT_VAAPI)
    {
        frames_ctx->sw_format = AV_PIX_FMT_NV12;

        if(hve_pixel_format_depth(h->sw_pix_fmt, &depth) != HVE_OK)
            return HVE_ERROR_MSG("failed to get pixel format depth");

        if(depth == 10)
            frames_ctx->sw_format = AV_PIX_FMT_P010LE;
    }

[bmegli]

Fixed