Support for HEIF / HEIC image transformations

joselcvarela commented 10 months ago

Describe the Improvement

In https://github.com/directus/directus/pull/20117 I have tried to add support for HEIC / HEIF image transformations. Although as mentioned by @paescuj and discussed previously on https://github.com/directus/directus/issues/17076 it's not as easy as just add mimetype to supported formats.

The goal of this post is to discuss how can we achieve the support for HEIC / HEIF image transformations.

joselcvarela commented 10 months ago

For the Docker image, we must need to include libs: libheif, libde265 and x265.

Support for patent-encumbered HEIC images using hevc compression requires the use of a globally-installed libvips compiled with support for libheif, libde265 and x265.

https://sharp.pixelplumbing.com/api-output#heif

DanielBiegler commented 9 months ago

Why this is harder than I thought initially

In order to make sharp use the needed libraries on your machine the libvips version must match the config inside of sharps package.json under config:

...
  "config": {
    "libvips": "8.14.5",
    "integrity": {
      "darwin-arm64v8": "sha512-1QZzICfCJd4wAO0P6qmYI5e5VFMt9iCE4QgefI8VMMbdSzjIXA9L/ARN6pkMQPZ3h20Y9RtJ2W1skgCsvCIccw==",
      "darwin-x64": "sha512-sMIKMYXsdU9FlIfztj6Kt/SfHlhlDpP0Ups7ftVFqwjaszmYmpI9y/d/q3mLb4jrzuSiSUEislSWCwBnW7MPTw==",
      "linux-arm64v8": "sha512-CD8owELzkDumaom+O3jJ8fKamILAQdj+//KK/VNcHK3sngUcFpdjx36C8okwbux9sml/T7GTB/gzpvReDrAejQ==",
      "linux-armv6": "sha512-wk6IPHatDFVWKJy7lI1TJezHGHPQut1wF2bwx256KlZwXUQU3fcVcMpV1zxXjgLFewHq2+uhyMkoSGBPahWzlA==",
      "linux-armv7": "sha512-HEZC9KYtkmBK5rUR2MqBhrVarnQVZ/TwLUeLkKq0XuoM2pc/eXI6N0Fh5NGEFwdXI2XE8g1ySf+OYS6DDi+xCQ==",
      "linux-x64": "sha512-SlFWrITSW5XVUkaFPQOySAaSGXnhkGJCj8X2wGYYta9hk5piZldQyMp4zwy0z6UeRu1qKTKtZvmq28W3Gnh9xA==",
      "linuxmusl-arm64v8": "sha512-ga9iX7WUva3sG/VsKkOD318InLlCfPIztvzCZKZ2/+izQXRbQi8VoXWMHgEN4KHACv45FTl7mJ/8CRqUzhS8wQ==",
      "linuxmusl-x64": "sha512-yeaHnpfee1hrZLok2l4eFceHzlfq8gN3QOu0R4Mh8iMK5O5vAUu97bdtxeZZeJJvHw8tfh2/msGi0qysxKN8bw==",
      "win32-arm64v8": "sha512-kR91hy9w1+GEXK56hLh51+hBCBo7T+ijM4Slkmvb/2PsYZySq5H7s61n99iDYl6kTJP2y9sW5Xcvm3uuXDaDgg==",
      "win32-ia32": "sha512-HrnofEbzHNpHJ0vVnjsTj5yfgVdcqdWshXuwFO2zc8xlEjA83BvXZ0lVj9MxPxkxJ2ta+/UlLr+CFzc5bOceMw==",
      "win32-x64": "sha512-BwKckinJZ0Fu/EcunqiLPwOLEBWp4xf8GV7nvmVuKKz5f6B+GxoA2k9aa2wueqv4r4RJVgV/aWXZWFKOIjre/Q=="
    },
    "runtime": "napi",
    "target": 7
  },
...

In order to get support for HEIF images, according to the sharp docs, we need a globally installed libvips that was compiled with needed dependencies. Per the docs and my local testing: If your local version doesnt match that specific version or above it will silently fall back to the vendored (read: downloaded prebuilt library) version.

As of right now (2023-11-30) we're using sharp@0.32.6 which lists libvips 8.14.5 as requirement.

Currently the stable v3.18 branch for alpine linux packages only supports both vips 8.14.3-r0 and vips-dev 8.14.3-r0 which means sharp always falls back to the vendored version 8.14.5 which is NOT compiled with support for HEIF images.

There is however the alpine linux edge branch for newer packages which indeed lists vips 8.14.5-r1 and since 2023-11-14 even 8.15.0-r0 which would allow us to rebuild the needed bindings via installing vips-dev and our needed libraries like libheif, libde265, x265, libjpeg, libpng, etc. before installing sharp itself so that it picks our custom libvips version.

This would work but turns out that alpine linux packages arent available forever. Example per the following alpine issue where this was detailed to break docker builds:

Example: on 2020 March 10th, I found gcc 9.2.0-r3 on the Alpine package repository (web UI) under branch 3.11. On 2020 March 23rd, just 13 days later, my Dockerfile failed to run because the package gcc 9.2.0-r3 had been revoked from the branch 3.11 of the package repository, and was replaced with gcc 9.2.0-r4.

This means if we install libvips and other dependencies from the package manager we might sooner or later run into the issue of breakage once sharp updates its own dependency of libvips and we have no control of pinning the other dependencies either for reproducible builds into the future.

One solution is to build libvips from source so that we can be sure we got the right thing for our needs. But this means we also gotta take care of including all of the needed and optional dependencies too. Here are some instructions for building but only Ubuntu, Debian, macOS, Windows, Cygwin so we'll gotta check out more specific instructions for our alpine based image which sometimes features different package names than other distros. We'll 100% have to customize the installation instructions and go from there.

Before we can build libvips then we need to correctly build libheif with its correct decoders and encoders, namely:

$ heif-convert --list-decoders
HEIC decoders:
- libde265 = libde265 HEVC decoder, version 1.0.9
AVIF decoders:
- dav1d = dav1d v6.6.0
- aom = AOMedia Project AV1 Decoder v3.5.0
$ heif-enc --list-encoders
HEIC encoders:
- x265 = x265 HEVC encoder (3.5+1-f0c1022b6) [default]
AVIF encoders:
- aom = AOMedia Project AV1 Encoder v3.5.0 [default]
- svt = SVT-AV1 encoder v1.1.0
- rav1e = Rav1e encoder

This seems like a heap of trouble for not enough gain if we cant find a way to skip all of this.

As far as I understood the alpine linux issue thread, over 2 year old branches arent updated from under your feet which would be neat but this then forces us to use an out of date sharp version and would block us from upgrading it when we want/need to.

Maybe theres hope (?)

Important to check out further: There is the comment of its author from literally yesterday where they overhauled the installation steps https://github.com/lovell/sharp/issues/3750

That might be helpful for us but I need to look into it further to see exactly what it means.

directus / directus