[Bug]: Encoding AV1 with color range "Full" outputs wrong colors

[akiirui]

Describe the bug

Encoding AV1 with color range "Full" outputs file has wrong color.

To Reproduce Steps to reproduce the behavior:

1. Open OBS
2. Set Output - "Video Encoder" to AMD HW AV1
3. Set Advanced - "Color Range" to limited. Equivalent to AMF_VIDEO_ENCODER_AV1_OUTPUT_COLOR_PROFILE=AMF_VIDEO_CONVERTER_COLOR_PROFILE_709.
4. Record a video with range limited
5. Set Advanced - "Color Range" to full. Equivalent to AMF_VIDEO_ENCODER_AV1_OUTPUT_COLOR_PROFILE=AMF_VIDEO_CONVERTER_COLOR_PROFILE_FULL_709.
6. Record a video with range full
7. Compare two videos

Setup (please complete the following information):

• OS: Windows 11
• Driver Version: 23.5.2
• GPU: RX 7900 XTX
• Which component has the issue: Encoder

Expected behavior

For the different color range "limited" and "full" the colors should be nearly identical when viewed with the naked eye.

Screenshots

Screenshot of limited:

Screenshot of full:

The original background image:

Additional context

mediainfo outputs of limited:

Video
ID                                       : 1
Format                                   : AV1
Format/Info                              : AOMedia Video 1
Format profile                           : Main@L5.2
Codec ID                                 : V_AV1
Duration                                 : 5 s 434 ms
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Original height                          : 1 082 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : sRGB/sYCC
Matrix coefficients                      : BT.709

mediainfo outputs of full:

Video
ID                                       : 1
Format                                   : AV1
Format/Info                              : AOMedia Video 1
Format profile                           : Main@L5.2
Codec ID                                 : V_AV1
Duration                                 : 5 s 400 ms
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Original height                          : 1 082 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Default                                  : No
Forced                                   : No
Color range                              : Full
colour_range_Original                    : Limited
Color primaries                          : BT.709
Transfer characteristics                 : sRGB/sYCC
Matrix coefficients                      : BT.709

example videos.zip

And in AMF_Video_Encode_AV1_API.md - Name: AMF_VIDEO_ENCODER_AV1_OUTPUT_COLOR_PROFILE:

Description: Color profile of the compressed output stream. SDR - Setting this parameter (COLOR_PROFILE) can fully describe a surface for SDR use case. HDR – For HDR use case the TRANSFER_CHARACTERISTIC, COLOR_PRIMARIES, and NOMINAL_RANGE parameters describe the surface. Determines the optional VUI parameter “matrix_coefficients”.

But NOMINAL_RANGE isn't available in VideoEncoderAV1.h

Original issue: https://github.com/obsproject/obs-studio/issues/9121

[MikhailAMD]

Could you please provide more details why you consider the result as an error? What I see is in "full" image black color is more deep compare to "limited" case. See details around thee dots in the text at left top corner. This is expected as for limited case color is compressed into limited range 16-235, while full is 0-255. MediaInfo seems showing the correct flags. More importantly, I believe, in OBS color conversion to YUV (where range is applied) is done in OBS by shaders, so AMF encoder just sets flags in VUI header.

[akiirui]

Please see "original background image", the full color range screenshot is blacker than original.

In a properly implemented color space, the player correctly displays limited and full as near-consistent colors.

And it differs from other implementations (AMF H264/H265 or SVT-AV1/AOM AV1).

mediainfo outputs of AOM-limited:

Video
ID                                       : 1
Format                                   : AV1
Format/Info                              : AOMedia Video 1
Format profile                           : Main@L4.1
Codec ID                                 : V_AV1
Duration                                 : 5 s 367 ms
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Default                                  : No
Forced                                   : No
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : sRGB/sYCC
Matrix coefficients                      : BT.709

mediainfo outputs of AOM-full:

Video
ID                                       : 1
Format                                   : AV1
Format/Info                              : AOMedia Video 1
Format profile                           : Main@L4.1
Codec ID                                 : V_AV1
Duration                                 : 5 s 384 ms
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 60.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Default                                  : No
Forced                                   : No
Color range                              : Full
Color primaries                          : BT.709
Transfer characteristics                 : sRGB/sYCC
Matrix coefficients                      : BT.709

Screenshot of limited from MPV:

Screenshot of full from MPV:

[akiirui]

And here have some test charts:

Video: test_charts.zip

Full:

Limited:

[flaeri]

More importantly, I believe, in OBS color conversion to YUV (where range is applied) is done in OBS by shaders, so AMF encoder just sets flags in VUI header.

I believe the issue lies with the tags/flags. When parsing the bitstream, the color range shows: seq_header_obu\color_config(): color_range 0 (should be 1 for full range)

Both FFmpeg, MPV and VLC reads the file as being partial range color (incorrect), hence the values being mishandeled, resulting in the full/partial mismatch.

[MikhailAMD]

I was confused by MediaInfo. It says "Full". Yes, both clips have color_range = 0. I've opened internal ticket. Thanks.

[cgutman]

@MikhailAMD any updates on the internal ticket? We're still seeing this behavior with 24.8.1 drivers.

In my local testing with the SimpleEncoder sample, it doesn't look like the color range information encoded in AMF_VIDEO_ENCODER_*_OUTPUT_COLOR_PROFILE is making it into the encoded bitstream on any codec. The specific nominal range properties for H.264 and HEVC (AMF_VIDEO_ENCODER_FULL_RANGE_COLOR and AMF_VIDEO_ENCODER_HEVC_NOMINAL_RANGE) do work and that's why full range is encoded correctly for those codecs in most applications. However, the AMF AV1 encoder doesn't have a nominal range option, which makes failure to derive the color range from the output color profile a major issue for encoding full range content.

I tested with the following patch and found that the output files were incorrectly written as if they were limited range content:

diff --git a/amf/public/samples/CPPSamples/SimpleEncoder/SimpleEncoder.cpp b/amf/public/samples/CPPSamples/SimpleEncoder/SimpleEncoder.cpp
index c035444..f1594b4 100644
--- a/amf/public/samples/CPPSamples/SimpleEncoder/SimpleEncoder.cpp
+++ b/amf/public/samples/CPPSamples/SimpleEncoder/SimpleEncoder.cpp
@@ -183,6 +183,9 @@ AMF_RESULT simpleEncode(const wchar_t* pCodec, const wchar_t* pFileNameOut) {

         res = encoder->SetProperty(AMF_VIDEO_ENCODER_FRAMESIZE, ::AMFConstructSize(widthIn, heightIn));
         AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_FRAMESIZE, %dx%d) failed", widthIn, heightIn);
+
+        res = encoder->SetProperty(AMF_VIDEO_ENCODER_OUTPUT_COLOR_PROFILE, AMF_VIDEO_CONVERTER_COLOR_PROFILE_FULL_709);
+        AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_OUTPUT_COLOR_PROFILE) failed");
     }
     else if (amf_wstring(pCodec) == amf_wstring(AMFVideoEncoder_HEVC))
     {
@@ -191,6 +194,9 @@ AMF_RESULT simpleEncode(const wchar_t* pCodec, const wchar_t* pFileNameOut) {

         res = encoder->SetProperty(AMF_VIDEO_ENCODER_HEVC_FRAMESIZE, ::AMFConstructSize(widthIn, heightIn));
         AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_HEVC_FRAMESIZE, %dx%d) failed", widthIn, heightIn);
+
+        res = encoder->SetProperty(AMF_VIDEO_ENCODER_HEVC_OUTPUT_COLOR_PROFILE, AMF_VIDEO_CONVERTER_COLOR_PROFILE_FULL_709);
+        AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_HEVC_OUTPUT_COLOR_PROFILE) failed");
     }
     else if (amf_wstring(pCodec) == amf_wstring(AMFVideoEncoder_AV1))
     {
@@ -199,6 +205,9 @@ AMF_RESULT simpleEncode(const wchar_t* pCodec, const wchar_t* pFileNameOut) {

         res = encoder->SetProperty(AMF_VIDEO_ENCODER_AV1_FRAMESIZE, ::AMFConstructSize(widthIn, heightIn));
         AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_AV1_FRAMESIZE, %dx%d) failed", widthIn, heightIn);
+
+        res = encoder->SetProperty(AMF_VIDEO_ENCODER_AV1_OUTPUT_COLOR_PROFILE, AMF_VIDEO_CONVERTER_COLOR_PROFILE_FULL_709);
+        AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_AV1_OUTPUT_COLOR_PROFILE) failed");
     }

     res = encoder->Init(formatIn, widthIn, heightIn);

[MikhailAMD]

OK, I checked, range value from all <>_COLOR_PROFILE parameters is not used for encoder VUI to avoid conflict with individual AMF_VIDEO_ENCODER_FULL_RANGE_COLOR and AMF_VIDEO_ENCODER_HEVC_NOMINAL_RANGE. It is used only if encoder does color conversion inside. I will discuss internally if this can be cleaned up. What is missing is AMF_VIDEO_ENCODER_AV1_NOMINAL_RANGE parameter to fill data in VUI for AV1. I will see If we can add it quicky.