Small tiles of encoded video doesn't decode probably of 2x1 tiles

[MartinEesmaa]

Hello, @nuomi2021!

When playing VVC video encoded of 2x1 tiles and maybe small tiles too and other tiles, the native VVC decoder doesn't probably correct decode well.

Source file can be found if you like to: https://media.xiph.org/video/derf/y4m/720p5994_stockholm_ter.y4m

Used to encode on my custom FFmpeg build:

ffmpeg_vvceasy -i 720p5994_stockholm_ter.y4m -c:v libvvenc -preset fast -qp 35 -vvenc-params tiles=2x1 720p5994_stockholm_ter.266

or same via vvenc with FFmpeg pipe

ffmpeg -i 720p5994_stockholm_ter.y4m -f yuv4mpegpipe - | vvencapp --y4m -i - --preset faster --qp 35 --tiles 2x1 -o 720p5994_stockholm_ter.266

External VVC decoder via libvvdec:

ffmpeg_vvceasy -c:v libvvdec -i 720p5994_stockholm_ter.266 -loglevel 8 -f md5 -
MD5=5114ea2645d99a758f978deb8af07ad4

Native VVC decoder on FFmpeg:

ffmpeg_vvceasy -i 720p5994_stockholm_ter.266 -loglevel 8 -f md5 -
MD5=084749641c464f533c1dce5ccf236658

Which means native VVC decoder is not probably decoded can cause random glitch artificial colors. I'll leave raw VVC bitstream here if you would like to download: 720p5994_stockholm_ter-vvc.zip

Result of native VVC decoder on FFmpeg on 3 seconds:

Also you can find more information when @AlisaFmla posted issue to my repository, same issue decoding 2x1 tiles on native VVC decoder: https://github.com/MartinEesmaa/VVCEasy/issues/42

I replied to him about the problem: https://github.com/MartinEesmaa/VVCEasy/issues/42#issuecomment-2003605280

Also thank you for your hard work. :)

Feel free to ask or reply to me.

Regards

• Martin Eesmaa

[frankplow]

Thanks for your report. I've managed to reproduce the issue and will have a look at it in the next few days.

FYI 2x1 tiles does not mean that the sequence is encoded using small blocks of 2x1 samples in some way, it means that the sequence is being divided into a 2x1 grid. This is why the distortion is constrained to the right half of the picture: the first tile (the left half) is being decoded fine and it is the second tile (the right half) which is problematic.

[MartinEesmaa]

No problem, @frankplow! I'm sure it may will fix in soon for few hours or days.

• Martin Eesmaa

[frankplow]

@nuomi2021 Been looking at this, the error seems to be in the inverse transform somewhere.

Can you please explain why the following is done at this particular location: https://github.com/ffvvc/FFmpeg/blob/e81b6d78fc2ddf8edd53a6a052713354ef8d27c2/libavcodec/vvc/vvc_cabac.c#L2291-L2292

sh_dep_quant_used_flag doesn't have an effect on the coeffs until the dequantisation stage in the spec, is this just applying some of its effects earlier in the decode process? The transform coeffs in VTM and FFVVC match until the line above, then differ. Also after dequantisation the two have different coefficients. It's not immediately clear how this interacts with tile partitioning, but I'll keep digging.

EDIT: nevermind, I misread the spec I see it does have some effect here.

[frankplow]

@nuomi2021 Been looking at this, the error seems to be in the inverse transform somewhere.

~Can you please explain why the following is done at this particular location:~

https://github.com/ffvvc/FFmpeg/blob/e81b6d78fc2ddf8edd53a6a052713354ef8d27c2/libavcodec/vvc/vvc_cabac.c#L2291-L2292

~sh_dep_quant_used_flag doesn't have an effect on the coeffs until the dequantisation stage in the spec, is this just applying some of its effects earlier in the decode process?~ The transform coeffs in VTM and FFVVC match until the line above, then differ. Also after dequantisation the two have different coefficients. It's not immediately clear how this interacts with tile partitioning, but I'll keep digging.

EDIT: nevermind, I misread the spec I see it does have some effect here.

@nuomi2021 Why do you think VTM does not have a condition on sh_dep_quant_used_flag here: https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/-/blob/master/source/Lib/DecoderLib/CABACReader.cpp#L3380

[frankplow]

Fix posted on upstream ML: https://patchwork.ffmpeg.org/project/ffmpeg/patch/20240319194751.34086-1-post@frankplowman.com/

[nuomi2021]

pushed https://github.com/FFmpeg/FFmpeg/commit/dfcf5f828df93df9f07c26fc39a010325522c120 thank you Frank and all.

Hi Frank, could you help put the 1 or 2frames to https://github.com/ffvvc/tests/tree/main/conformance/passed/corner_cases you can use

ffmpeg -i input.vvc -c:v copy -vframes 1 outpu.vvc

[birdie-github]

Must be fixed in git master: https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/dfcf5f828df93df9f07c26fc39a010325522c120

[nuomi2021]

thank you all for your help on this.