search

strukturag / libheif

libheif is an HEIF and AVIF file format decoder and encoder.
Other
1.77k stars 307 forks source link

Add support for smooth chroma upsampling #586

Open maryla-uc opened 2 years ago

maryla-uc commented 2 years ago

Libheif currently only supports nearest neighbor chroma upsampling, which often leads to extra artifacts when decoding YUV420 images. Most codecs support a bilinear chroma upsampling with weights 9:3:3:1 E.g. if you have the following Luma and Chroma samples (with centered chroma samples, as per issue #521):

L   L  L    L
  C1     C2
L   L*  L   L 

L   L  L    L
  C3     C4
L   L  L    L

then the value of chroma at the position marked with would be: C = (9 C1 + 3 C2 + 3 C3 + C4)/16

See also the implementation in libavif: https://github.com/AOMediaCodec/libavif/blob/ded15fb7b41b0513e138ba457034a2fb91c1f2c9/src/reformat.c#L496

Here is an example (artificial) avif image, decoded with libheif (nearest neighbor) and with libavif (smooth upsampling). The difference is very obvious when zooming in.

C95FUpghqahiGyo-1

farindk commented 2 years ago

Thanks for the proposal. That filter seems easy to implement. I could add a simple implementation to libheif. For an accelerated (SIMD) version, I would prefer to reuse an external implementation (e.g. libyuv).

I am wondering what would be the best way to choose the filters. In theory, the upsampling filter should match the downsampling filter, but that is not known to the decoder. Signaling the preferred upload filter might be something that could be added to the file formats. Until then, we probably have to pass the decision onto the user.

maryla-uc commented 2 years ago

The YUV420 to 444 conversion function in libyuv do use bilinear upsampling so if you want to use that, it's indeed an option. Unfortunately, libyuv's YUV420 to RGB conversion does NOT currently use bilinear, but simple nearest neighbor upsampling. See https://bugs.chromium.org/p/libyuv/issues/detail?id=872 Until this gets fixed, the workaround is to convert in two steps: YUV420->YUV444->RGB.

I agree users should be able to choose the filter applied, with a sensible default set otherwise. I think the default should be bilinear.