[Bug]: DecoderConfigurationRecord is required to decode resolutions with frame cropping

[RFCArrow]

Describe the bug On input resolutions where a conformance window (frame cropping) must be applied, the decoder does not function correctly unless an AVCDecoderConfigurationRecord is supplied in the AMF_VIDEO_DECODER_EXTRADATA property.

An example resolution affected by this is 1920x1080 which will be cropped down from 1920x1088.

This bug effects both H264 and HEVC.

To Reproduce Apply the following patch to SimpleDecoder.cpp

diff --git a/amf/public/samples/CPPSamples/SimpleDecoder/SimpleDecoder.cpp b/amf/public/samples/CPPSamples/SimpleDecoder/SimpleDecoder.cpp
index 222cc82..cf3a7b7 100644
--- a/amf/public/samples/CPPSamples/SimpleDecoder/SimpleDecoder.cpp
+++ b/amf/public/samples/CPPSamples/SimpleDecoder/SimpleDecoder.cpp
@@ -62,7 +62,7 @@ static void WritePlane(amf::AMFPlane *plane, FILE *f);
 static void WaitDecoder(amf::AMFContext *context, amf::AMFSurface *surface); // Waits till decoder finishes decode the surface. Need for accurate profiling only. Do not use in the product!!!

 // The memory transfer from DX9 to HOST and writing a raw file is longer than decode time. To measure decode time correctly disable convert and write here: 
-static bool bWriteToFile = false;
+static bool bWriteToFile = true;
 #define START_TIME_PROPERTY L"StartTimeProperty" // custom property ID to store submission time in a frame - all custom properties are copied from input to output

 #define MILLISEC_TIME     10000
@@ -171,15 +171,8 @@ int main(int argc, char* argv[])
                res = g_AMFFactory.GetFactory()->CreateComponent(context, AMFVideoDecoderHW_H265_HEVC, &decoder);

     res = decoder->SetProperty(AMF_TIMESTAMP_MODE, amf_int64(AMF_TS_DECODE)); // our sample H264 parser provides decode order timestamps - change this depend on demuxer
+       parser->SetUseStartCodes(true);

-    if (parser->GetExtraDataSize()) 
-    { // set SPS/PPS extracted from stream or container; Alternatively can use parser->SetUseStartCodes(true)
-        amf::AMFBufferPtr buffer;
-        context->AllocBuffer(amf::AMF_MEMORY_HOST, parser->GetExtraDataSize(), &buffer);
-
-        memcpy(buffer->GetNative(), parser->GetExtraData(), parser->GetExtraDataSize());
-        decoder->SetProperty(AMF_VIDEO_DECODER_EXTRADATA, amf::AMFVariant(buffer));
-    }
     res = decoder->Init(formatOut, parser->GetPictureWidth(), parser->GetPictureHeight());

     PollingThread thread(context, decoder, fileNameOutWidthSize);

Then build and run the SimpleDecoder project: cd AMF/amf VK_SDK_PATH=~/vulkan/1.3.239.0 make -C public/samples ./bin/dbg_64/SimpleDecoder input_1920x1080.h264

Inspect the output file output_1920x1080.nv12 and observe that I frames feature an area of garbage pixels at the bottom of the frame. Also observe that the P frames are completely blank.

Note that the effect is not observed when the resolution is a whole number of macroblocks, for example 3840x2160.

Setup (please complete the following information): OS: Ubuntu Linux 22.04 LTS (5.19.0-45-generic) GPU: RX 7900 XTX Driver Version:

amdgpu         1:5.4.50406-1580598.22.04 amd64        Meta package to install amdgpu components.
libamdenc-amdgpu-pro:amd64 1.0-1580631.22.04 amd64        AMD Encode Core Library
amf-amdgpu-pro:amd64 1.4.29-1580631.22.04 amd64        AMDGPU Pro Advanced Multimedia Framework

AMF Source: 1.4.29 origin/master (bd5db31) Which component has the issue: Encoder

Expected behavior All frames are correctly decoded in the output file.

Additional context While decoding HEVC bitstreams, setting the AMF_VIDEO_DECODER_EXTRADATA seems to require that the input bitstream is provided in AVCC format instead of Annex B format. If the input bitstream is in Annex B format while the AMF_VIDEO_DECODER_EXTRADATA property is set, it tends to lead to the decoder segfaulting within libamfrt64.so. While this behaviour makes some sense, this requirement does not appear to be documented anywhere in the documentation. Note that this behaviour is not observed while decoding H264 bitstreams.

While investigating this bug, I noticed that the comments in BitStreamParserH265.cpp:GetExtradata() incorrectly reference AVCDecoderConfigurationRecord instead of HEVCDecoderConfigurationRecord. In particular, the section reference in ISO-IEC 14496-15 should be #8.3.3 and numSequenceParameterSets should not be mentioned.

[RFCArrow]

For convenience, here is a screenshot of the effect: and here is a short sample of the nv12 output: short_960x540_nv12.zip

[rhutsAMD]

In the case where the P-frames are blank, is extradata set? If extradata is set then are the P-frames no longer blank?

[RFCArrow]

"In the case where the P-frames are blank, is extradata set?"

No

"If extradata is set then are the P-frames no longer blank?"

Yes, except for on HEVC when UseStartCodes is set, in which case the decoder crashes

I have summarised the effect in the table below. For your benefit, here are the input elementary streams used sample_input.zip and the first few frames of the output for each case which generates output sample_output.zip.

Codec	Extradata	UseStartCodes	Outcome
H264	Set	False	Video renders correctly
H264	Unset	False	Output file is empty
H264	Set	True	Video renders correctly
H264	Unset	True	Video has garbage pixels and blank P-frames
HEVC	Set	False	Video renders correctly
HEVC	Unset	False	SimpleDecoder segfaults in AMFh265Parser_Fast::ParseSliceHeader()
HEVC	Set	True	SimpleDecoder segfaults in AMFh265Parser_Fast::DecodeBuffer()
HEVC	Unset	True	Video has garbage pixels

[RFCArrow]

Hey, is there any update on this? Have you managed to reproduce the issue?

[rhutsAMD]

Yes the issue was reproduced. We will provide another update once we have some fixes.

[RFCArrow]

Great! Thanks for the update :+1:

[MemeTao]

Is there any updates?

[rhutsAMD]

The 4 valid cases of combinations of Extradata and UseStartCodes are below. Only one of Extradata and UseStartCodes should be enabled at a time, and not both or neither.

There have been some fixes such as for a crash with particular inputs on HEVC when both Extradata and UseStartCodes are unset, but this is an invalid option combination to use regardless.

Codec	Extradata	UseStartCodes
H264	Set	FALSE
H264	Unset	TRUE
HEVC	Set	FALSE
HEVC	Unset	TRUE