Lib Heif still not available after loading the lambda layer

Hi,

First of all, thank you for this project. I spent dozens of hours trying to convert a HEIC file into JPG in my lambda function and I didn't find anything relevant.

I was able to use your Dockerfile to generate the nodejs10 compatible libs and I was also able to deploy my function with the new layer (deployed based on your project). In the lambda function, requiring sharp works but when I try to convert the image, I get the same error message as if it was the version of sharp not compiled with HEIC:

Error: Input buffer contains unsupported image format

To validate that lambda is using the right custom version of sharp, I removed the layer attached to my function and sharp is not found anymore, so the version used is the good one from the layer.

Printing sharp.format:

sharp format: { jpeg: { id: 'jpeg', input: { file: true, buffer: true, stream: true }, output: { file: true, buffer: true, stream: true } }, png: { id: 'png', input: { file: true, buffer: true, stream: true }, output: { file: true, buffer: true, stream: true } }, webp: { id: 'webp', input: { file: true, buffer: true, stream: true }, output: { file: true, buffer: true, stream: true } }, tiff: { id: 'tiff', input: { file: true, buffer: true, stream: true }, output: { file: true, buffer: true, stream: true } }, magick: { id: 'magick', input: { file: false, buffer: false, stream: false }, output: { file: false, buffer: false, stream: false } }, openslide: { id: 'openslide', input: { file: false, buffer: false, stream: false }, output: { file: false, buffer: false, stream: false } }, dz: { id: 'dz', input: { file: false, buffer: false, stream: false }, output: { file: true, buffer: true, stream: true } }, ppm: { id: 'ppm', input: { file: false, buffer: false, stream: false }, output: { file: false, buffer: false, stream: false } }, fits: { id: 'fits', input: { file: false, buffer: false, stream: false }, output: { file: false, buffer: false, stream: false } }, gif: { id: 'gif', input: { file: true, buffer: true, stream: true }, output: { file: false, buffer: false, stream: false } }, svg: { id: 'svg', input: { file: true, buffer: true, stream: true }, output: { file: false, buffer: false, stream: false } }, heif: { id: 'heif', input: { file: false, buffer: false, stream: false }, output: { file: false, buffer: false, stream: false } }, pdf: { id: 'pdf', input: { file: false, buffer: false, stream: false }, output: { file: false, buffer: false, stream: false } }, vips: { id: 'vips', input: { file: true, buffer: false, stream: false }, output: { file: true, buffer: false, stream: false } }, raw: { id: 'raw', input: { file: false, buffer: true, stream: true }, output: { file: false, buffer: true, stream: true } } }

and sharp.versions:

sharp versions: { cairo: '1.16.0', exif: '0.6.22', expat: '2.2.9', ffi: '3.3', fontconfig: '2.13.92', freetype: '2.10.2', fribidi: '1.0.10', gdkpixbuf: '2.40.0', gettext: '0.21', gif: '5.1.4', glib: '2.65.2', gsf: '1.14.47', harfbuzz: '2.7.1', jpeg: '2.0.5', lcms: '2.11', orc: '0.4.31', pango: '1.46.1', pixman: '0.40.0', png: '1.6.37', svg: '2.49.4', spng: '0.6.0', tiff: '4.1.0', vips: '8.10.0', webp: '1.1.0', xml: '2.9.10', zlib: '1.2.11' }

It seems that HEIF / HEIC libs are not taken into consideration by sharp.

I wanted to validate that the files are available so I printed the content of /opt/lib directory:

path: '/opt/lib/', name: 'lib', children: [ { path: '/opt/lib/libICE.so.6', name: 'libICE.so.6', size: 104544, extension: '.6', type: 'file' }, { path: '/opt/lib/libSM.so.6', name: 'libSM.so.6', size: 32256, extension: '.6', type: 'file' }, { path: '/opt/lib/libX11.so.6', name: 'libX11.so.6', size: 1294096, extension: '.6', type: 'file' }, { path: '/opt/lib/libXau.so.6', name: 'libXau.so.6', size: 15432, extension: '.6', type: 'file' }, { path: '/opt/lib/libc.so.6', name: 'libc.so.6', size: 2021416, extension: '.6', type: 'file' }, { path: '/opt/lib/libde265.so', name: 'libde265.so', size: 10731296, extension: '.so', type: 'file' }, { path: '/opt/lib/libde265.so.0', name: 'libde265.so.0', size: 10731296, extension: '.0', type: 'file' }, { path: '/opt/lib/libde265.so.0.0.10', name: 'libde265.so.0.0.10', size: 10731296, extension: '.10', type: 'file' }, { path: '/opt/lib/libdl.so.2', name: 'libdl.so.2', size: 19208, extension: '.2', type: 'file' }, { path: '/opt/lib/libexif.so.12', name: 'libexif.so.12', size: 287896, extension: '.12', type: 'file' }, { path: '/opt/lib/libexpat.so.1', name: 'libexpat.so.1', size: 197984, extension: '.1', type: 'file' }, { path: '/opt/lib/libffi.so.6', name: 'libffi.so.6', size: 32240, extension: '.6', type: 'file' }, { path: '/opt/lib/libgcc_s.so.1', name: 'libgcc_s.so.1', size: 88640, extension: '.1', type: 'file' }, { path: '/opt/lib/libgif.so.4', name: 'libgif.so.4', size: 42000, extension: '.4', type: 'file' }, { path: '/opt/lib/libglib-2.0.so.0', name: 'libglib-2.0.so.0', size: 1123592, extension: '.0', type: 'file' }, { path: '/opt/lib/libgmodule-2.0.so.0', name: 'libgmodule-2.0.so.0', size: 15536, extension: '.0', type: 'file' }, { path: '/opt/lib/libgobject-2.0.so.0', name: 'libgobject-2.0.so.0', size: 334784, extension: '.0', type: 'file' }, { path: '/opt/lib/libheif.so', name: 'libheif.so', size: 6121000, extension: '.so', type: 'file' }, { path: '/opt/lib/libheif.so.1', name: 'libheif.so.1', size: 6121000, extension: '.1', type: 'file' }, { path: '/opt/lib/libheif.so.1.3.2', name: 'libheif.so.1.3.2', size: 6121000, extension: '.2', type: 'file' }, { path: '/opt/lib/libjbig.so.2.0', name: 'libjbig.so.2.0', size: 51312, extension: '.0', type: 'file' }, { path: '/opt/lib/libjpeg.so.62', name: 'libjpeg.so.62', size: 272904, extension: '.62', type: 'file' }, { path: '/opt/lib/libm.so.6', name: 'libm.so.6', size: 1414728, extension: '.6', type: 'file' }, { path: '/opt/lib/libpcre.so.1', name: 'libpcre.so.1', size: 410400, extension: '.1', type: 'file' }, { path: '/opt/lib/libpng15.so.15', name: 'libpng15.so.15', size: 171016, extension: '.15', type: 'file' }, { path: '/opt/lib/libpthread.so.0', name: 'libpthread.so.0', size: 149272, extension: '.0', type: 'file' }, { path: '/opt/lib/libstdc++.so.6', name: 'libstdc++.so.6', size: 1566264, extension: '.6', type: 'file' }, { path: '/opt/lib/libtiff.so.5', name: 'libtiff.so.5', size: 471184, extension: '.5', type: 'file' }, { path: '/opt/lib/libuuid.so.1', name: 'libuuid.so.1', size: 20056, extension: '.1', type: 'file' }, { path: '/opt/lib/libvips-cpp.so', name: 'libvips-cpp.so', size: 1539312, extension: '.so', type: 'file' }, { path: '/opt/lib/libvips-cpp.so.42', name: 'libvips-cpp.so.42', size: 1539312, extension: '.42', type: 'file' }, { path: '/opt/lib/libvips-cpp.so.42.11.2', name: 'libvips-cpp.so.42.11.2', size: 1539312, extension: '.2', type: 'file' }, { path: '/opt/lib/libvips.so', name: 'libvips.so', size: 11916416, extension: '.so', type: 'file' }, { path: '/opt/lib/libvips.so.42', name: 'libvips.so.42', size: 11916416, extension: '.42', type: 'file' }, { path: '/opt/lib/libvips.so.42.11.2', name: 'libvips.so.42.11.2', size: 11916416, extension: '.2', type: 'file' }, { path: '/opt/lib/libwebp.so', name: 'libwebp.so', size: 3578976, extension: '.so', type: 'file' }, { path: '/opt/lib/libwebp.so.7', name: 'libwebp.so.7', size: 3578976, extension: '.7', type: 'file' }, { path: '/opt/lib/libwebp.so.7.0.4', name: 'libwebp.so.7.0.4', size: 3578976, extension: '.4', type: 'file' }, { path: '/opt/lib/libwebpdemux.so', name: 'libwebpdemux.so', size: 89000, extension: '.so', type: 'file' }, { path: '/opt/lib/libwebpdemux.so.2', name: 'libwebpdemux.so.2', size: 89000, extension: '.2', type: 'file' }, { path: '/opt/lib/libwebpdemux.so.2.0.6', name: 'libwebpdemux.so.2.0.6', size: 89000, extension: '.6', type: 'file' }, { path: '/opt/lib/libwebpmux.so', name: 'libwebpmux.so', size: 208488, extension: '.so', type: 'file' }, { path: '/opt/lib/libwebpmux.so.3', name: 'libwebpmux.so.3', size: 208488, extension: '.3', type: 'file' }, { path: '/opt/lib/libwebpmux.so.3.0.4', name: 'libwebpmux.so.3.0.4', size: 208488, extension: '.4', type: 'file' }, { path: '/opt/lib/libxcb.so.1', name: 'libxcb.so.1', size: 165784, extension: '.1', type: 'file' }, { path: '/opt/lib/libz.so.1', name: 'libz.so.1', size: 85976, extension: '.1', type: 'file' } ]

I can see that Libvips and libheif.so.1 are present. Do you have any idea why it's not including the right libs? Is there any path to specify?

Thank you

Created a Dockerfile based on the repo one. It supports heic. It's a lambda layer which will provide sharp so make sure to not include sharp in the dependencies of your lambda.

FROM lambci/lambda:build-nodejs12.x as base-box

RUN yum update -y \
    && yum groupinstall -y "Development Tools" \
    && yum install -y wget tar nano

FROM base-box as build-stage

# libvips dependencies
RUN yum install -y --setopt=skip_missing_names_on_install=False \
    libpng-devel \
    glib2-devel \
    libjpeg-devel \
    libjpeg-turbo-devel \
    zlib-devel \
    expat-devel \
    libtiff-devel \
    giflib-devel \
    librsvg2-devel \
    libwebp-devel \
    poppler-devel \
    poppler-glib-devel \
    libexif-devel

ENV VIPSHOME /opt
ENV PKG_CONFIG_PATH $VIPSHOME/lib/pkgconfig

ARG WEBP_VERSION=1.2.0
ARG WEBP_URL=https://storage.googleapis.com/downloads.webmproject.org/releases/webp

RUN cd /usr/local/src \
    && wget ${WEBP_URL}/libwebp-${WEBP_VERSION}.tar.gz \
    && tar xzf libwebp-${WEBP_VERSION}.tar.gz \
    && cd libwebp-${WEBP_VERSION} \
    && ./configure --enable-libwebpmux --enable-libwebpdemux --enable-libwebpdecoder \
    --prefix=$VIPSHOME \
    && make V=0 \
    && make install

ARG LIBDE265_VERSION=1.0.8
ARG LIBDE265_URL=https://github.com/strukturag/libde265/releases/download/v${LIBDE265_VERSION}

RUN cd /usr/local/src \
    && wget ${LIBDE265_URL}/libde265-${LIBDE265_VERSION}.tar.gz \
    && tar xzf libde265-${LIBDE265_VERSION}.tar.gz \
    && cd libde265-${LIBDE265_VERSION} \
    && ./configure --disable-dec265 --disable-sherlock265 \
    --prefix=$VIPSHOME \
    && make V=0 \
    && make install

ARG LIBHEIF_VERSION=1.11.0
ARG LIBHEIF_URL=https://github.com/strukturag/libheif/releases/download/v${LIBHEIF_VERSION}

RUN cd /usr/local/src \
    && wget ${LIBHEIF_URL}/libheif-${LIBHEIF_VERSION}.tar.gz \
    && tar xzf libheif-${LIBHEIF_VERSION}.tar.gz \
    && cd libheif-${LIBHEIF_VERSION} \
    && ./configure \
    --prefix=$VIPSHOME \
    && make \
    && make install

ARG VIPS_VERSION=8.10.5
ARG VIPS_URL=https://github.com/libvips/libvips/releases/download

RUN cd /usr/local/src \
    && wget ${VIPS_URL}/v${VIPS_VERSION}/vips-${VIPS_VERSION}.tar.gz \
    && tar xzf vips-${VIPS_VERSION}.tar.gz \
    && cd vips-${VIPS_VERSION} \
    && ./configure --prefix=$VIPSHOME \
    && make \
    && make install 

WORKDIR /etc/ld.so.conf.d
RUN echo "/opt/lib" >> libvips.conf && ldconfig -v

WORKDIR /var/task

RUN mkdir -p lib
WORKDIR /var/task/lib

RUN cp -a $VIPSHOME/lib/*.so* . && \
    cp /usr/lib64/libgobject-2.0.so.0 . && \
    cp /usr/lib64/libglib-2.0.so.0 . && \
    cp /usr/lib64/libstdc++.so.6 . && \
    cp /usr/lib64/libm.so.6 . && \
    cp /usr/lib64/libgcc_s.so.1 . && \
    cp /usr/lib64/libpthread.so.0 . && \
    cp /usr/lib64/libc.so.6 . && \
    cp /usr/lib64/libz.so.1 . && \
    cp /usr/lib64/libgmodule-2.0.so.0 . && \
    cp /usr/lib64/libgif.so.4 . && \
    cp /usr/lib64/libpng15.so.15 . && \
    cp /usr/lib64/libtiff.so.5 . && \
    cp /usr/lib64/libjpeg.so.62 . && \
    cp /usr/lib64/libexpat.so.1 . && \
    cp /usr/lib64/libexif.so.12 . && \
    cp /usr/lib64/libjbig.so.2.0 . && \
    cp /usr/lib64/libICE.so.6 . && \
    cp /usr/lib64/libSM.so.6 . && \
    cp /usr/lib64/libxcb.so.1 . && \
    cp /usr/lib64/libXau.so.6 . && \
    cp /usr/lib64/libX11.so.6 . && \
    cp /usr/lib64/libgdk_pixbuf-2.0.so.0 . && \
    cp /usr/lib64/libcairo.so.2 . && \
    cp /usr/lib64/libpangocairo-1.0.so.0 . && \
    cp /usr/lib64/libpangoft2-1.0.so.0 . && \
    cp /usr/lib64/libpango-1.0.so.0 . && \
    cp /usr/lib64/libfontconfig.so.1 . && \
    cp /usr/lib64/libfreetype.so.6 . && \
    cp /usr/lib64/libcroco-0.6.so.3 . && \
    cp /usr/lib64/libpixman-1.so.0 . && \
    cp /usr/lib64/libEGL.so.1 . && \
    cp /usr/lib64/libxcb-shm.so.0 . && \
    cp /usr/lib64/libxcb-render.so.0 . && \
    cp /usr/lib64/libXrender.so.1 . && \
    cp /usr/lib64/libXext.so.6 . && \
    cp /usr/lib64/libGL.so.1 . && \
    cp /usr/lib64/libharfbuzz.so.0 . && \
    cp /usr/lib64/libthai.so.0 . && \
    cp /usr/lib64/libfribidi.so.0 . && \
    cp /usr/lib64/libGLdispatch.so.0 . && \
    cp /usr/lib64/libGLX.so.0 . && \
    cp /usr/lib64/libgraphite2.so.3 . && \
    cp /usr/lib64/libpoppler.so.46 . && \
    cp /usr/lib64/libpoppler-glib.so.18 . && \
    cp /usr/lib64/liblcms2.so.2 . && \
    cp /usr/lib64/libopenjpeg.so.1 . && \
    cp /usr/lib64/libxml2.so.2 . && \
    cp /usr/lib64/libbz2.so.1 . && \
    cp /usr/lib64/libuuid.so.1 . && \
    cp /usr/lib64/libgio-2.0.so.0 . && \
    cp /usr/lib64/liblzma.so.5 . && \
    cp /usr/lib64/libmount.so.1 . && \
    cp /usr/lib64/libblkid.so.1 . && \
    cp /usr/lib64/librsvg-2.so.2 . 

WORKDIR /var/task

RUN mkdir -p nodejs && cd nodejs && npm install sharp

FROM scratch as export-stage

COPY --from=build-stage /var/task/lib/. dist/lib/.
COPY --from=build-stage /var/task/nodejs/node_modules dist/nodejs/node_modules

Build it with: DOCKER_BUILDKIT=1 docker build -f Dockerfile -o . . It will produce a dist folder which is the lambda layer.

cd dist && zip -r nodejs lib

Hope it helps!

smcguinness / SharpLayer

Lib Heif still not available after loading the lambda layer #3