[Bug]: HEVC encoder fails if pre-analysis is enabled with 10-bit output

[Chipcraft]

The issue occurs when pre-analysis is enabled either by choice, or automatically per the requirement of the selected rate-control mode together with the 10-bit output color-depth. Everything works as intended (considering...), when the output color-depth is 8-bits.

The official "TranscodeHW" sample from AMF 1.4.29 exits without producing any output bitstream when 10-bit color-depth (-HevcColorBitDepth 10) is specified regardless of the other parameters, so it is rather hard to cross-reference these results with that.

Radeon RX 7900 XTX w/ Adrenalin 23.5.2 (31.0.14057.5006) Windows 10 PRO 22H2 19045.3086

HEVC_VBR_420_NOPA.log = PEAK_CONSTRAINED_VBR, MAIN, YUV420P, no pre-analysis by choice. // OK HEVC_VBR_P010_NOPA.log = PEAK_CONSTRAINED_VBR, MAIN_10, P010LE, no pre-analysis by choice. // OK HEVC_VBR_420_PA.log = PEAK_CONSTRAINED_VBR, MAIN, YUV420P, pre-analysis enabled by choice. // OK HEVC_VBR_P010_PA.log = PEAK_CONSTRAINED_VBR, MAIN_10, P010LE, pre-analysis enabled by choice. // FAIL

It makes no difference if pre-analysis is enabled by choice, or automatically per the requirement of the selected rate-control mode. Therefore PEAK_CONSTRAINED_VBR will fail with the same exact way as the rate-control modes which require and automatically enable it (QUALITY_VBR, HIGH_QUALITY_VBR, HIGH_QUALITY_CBR).

HEVC_VBR_420_NOPA.log HEVC_VBR_420_PA.log HEVC_VBR_P010_NOPA.log HEVC_VBR_P010_PA.log

[rhutsAMD]

The AMF PA component currently only accepts raw input images in the NV12 format. AMF_Video_PreAnalysis_API.md

The HEVC + P010 + PA case shows it is unsupported in the last attached log.

We will add more error checking in AMF.

[Chipcraft]

Frankly, being limited to just 8-bit is a pretty brutal limitation when it comes to HEVC and AV1, as using anything but 10-bit makes very little sense and at least in case of HEVC is rarely even done.

Quality and feature wise AMD's encoders are still behind the competition as is, so the new rate control modes (QVBR, HQVBR, HQCBR) requiring the pre-analysis being limited to just 8-bit output is not only sad, but seems like a wasted opportunity to close the gap to the competition both in terms of quality and features. Both Intel (QSV: ICQ) and NVIDIA (NVENC: CQ VBR) have had similar, more advanced rate control methods available for years and obviously, without being limited to 8-bit color depth.

Surely there are plans, or better yet work being done to get rid of the NV12 input requirement / limitation of pre-analysis?

[cs9kc]

I agree! You aren't the first to request the capability of the PreAnalysis Component to support 10-bit HDR encoding

The PA Component does quite a bit to improve encoding quality in SDR content, and it would be a welcome addition to have 10-bit HDR support.

[rhutsAMD]

Additional error checking has been added and will be available in a future public driver release. Attempting to initialize the AMF encoder with P010 format and PA enabled is not a valid configuration.

Regarding 10-bit support for PA, please stay tuned.