sharp custom libvips issue

atxplangl commented 2 years ago

What are you trying to achieve?

I'm trying to package sharp with custom libvips. Before I add poppler and it's dependencies I want to get a baseline. Everything builds and compiles, but when I look at node_modules/sharp/vender it shows 8.13.2 and non custom versions even though I'm building is v8.10.2. I don't mind updating versions to latest, but I feel I'm missing a required flag to pull custom libvips. One thing to note is I set config.libvips to 8.10.2 in my package.json and validated pkg-config --modversion vips-cpp is set to 8.10.0 on container. Any advice on what I'm missing would be great. Dockerfile and package.json used in Dockerfile is listed below

When you searched for similar issues, what did you find that might be related?

Please provide a minimal, standalone code sample, without other dependencies, that demonstrates this question

Dockerfile



WORKDIR /build

ARG VIPS_VERSION=8.10.0

ENV VIPS_VERSION=$VIPS_VERSION
ENV PATH=/opt/bin:$PATH
ENV LD_LIBRARY_PATH=/opt/lib:/opt/lib64:$LD_LIBRARY_PATH
ENV PKG_CONFIG_PATH=/opt/lib/pkgconfig:/opt/lib64/pkgconfig
ENV CFLAGS="-fexceptions -Wall -O3"
ENV CXXFLAGS="${CFLAGS}"

# Setup Some Dirs
#
RUN mkdir -p \
    share/lib \
    share/include
# Install expat
#
RUN curl https://codeload.github.com/libexpat/libexpat/zip/R_2_2_9 > libexpat-R_2_2_9.zip && \
    unzip libexpat-R_2_2_9.zip && \
    cd ./libexpat-R_2_2_9/expat && \
    ./buildconf.sh && \
    ./configure --prefix=/opt && \
    make install

RUN cp -a /opt/lib/libexpat.so* /build/share/lib

# Install libpng
#
RUN curl -L https://fossies.org/linux/misc/libpng-1.6.38.tar.xz > libpng-1.6.38.tar.gz && \
     tar -xf libpng-1.6.38.tar.gz && \
     cd libpng-1.6.38 && \
     ./configure --prefix=/opt --disable-static && \
     make && \
     make install

RUN cp -a /opt/lib/libpng.so* /build/share/lib && \
    cp -a /opt/lib/libpng16.so* /build/share/lib

# Install giflib
#
RUN curl -L https://fossies.org/linux/misc/giflib-5.2.1.tar.gz > giflib-5.2.1.tar.gz && \
    tar -xf giflib-5.2.1.tar.gz && \
    cd giflib-5.2.1 && \
    make && \
    make PREFIX=/opt install

RUN cp -a /opt/lib/libgif.so* /build/share/lib

# Install libjpeg-turbo
#
RUN curl -L https://fossies.org/linux/misc/libjpeg-turbo-2.1.4.tar.gz > libjpeg-turbo-2.1.4.tar.gz && \
    tar -xf libjpeg-turbo-2.1.4.tar.gz && \
    cd libjpeg-turbo-2.1.4 && \
    cmake -DCMAKE_INSTALL_PREFIX=/opt && \
    make && \
    make install

RUN cp -a /opt/lib64/libjpeg.so* /build/share/lib && \
    cp -a /opt/lib64/libturbojpeg.so* /build/share/lib

# Install libimagequant
#
RUN git clone https://github.com/ImageOptim/libimagequant.git && \
    cd ./libimagequant && \
    git checkout 2.12.6 && \
    ./configure --prefix=/opt && \
    make libimagequant.so && \
    make install && \
    echo /opt/lib > /etc/ld.so.conf.d/libimagequant.conf && \
    ldconfig

RUN cp -a /opt/lib/libimagequant.so* /build/share/lib/

# Install libfftw
#
RUN curl -L http://www.fftw.org/fftw-3.3.8.tar.gz > fftw-3.3.8.tar.gz && \
    tar -xf fftw-3.3.8.tar.gz && \
    cd ./fftw-3.3.8 && \
    ./configure \
      --prefix=/opt \
      --enable-shared \
      --disable-static \
      --enable-threads \
      --enable-sse2 \
      --enable-avx && \
    make && \
    make install

RUN cp -a /opt/lib/libfftw3* /build/share/lib/

# Install liborc (perf)
#
RUN curl -Lk https://gstreamer.freedesktop.org/data/src/orc/orc-0.4.26.tar.xz > orc-0.4.26.tar.xz && \
    tar -xf orc-0.4.26.tar.xz && \
    cd orc-0.4.26 && \
    ./configure --prefix=/opt && \
    make && \
    make install
RUN cp -a /opt/lib/liborc-0.4.so* /build/share/lib/

# Install libvips. Primary deps https://libvips.github.io/libvips/install.html
#
RUN yum install -y \
    gtk-doc \
    ninja-build \
    wget \
    libffi-devel \
    gobject-introspectioni

# RUN yum remove -y cmake 
# RUN wget https://cmake.org/files/v3.10/cmake-3.10.0.tar.gz
# RUN tar -xvzf cmake-3.10.0.tar.gz;cd cmake-3.10.0;./bootstrap && make && make install

RUN pip3 install meson && \
    pip3 install ninja

RUN cp -a /usr/lib64/libgio-2.0* /build/share/lib && \
    cp -a /usr/lib64/libglib-2.0.so* /build/share/lib && \
    cp -a /usr/lib64/libgmodule-2.0.so* /build/share/lib && \
    cp -a /usr/lib64/libgobject-2.0.so* /build/share/lib && \
    cp -a /usr/lib64/libgthread-2.0.so* /build/share/lib && \ 
    cp -a /usr/lib64/libffi.so* /build/share/lib

RUN curl -Lk http://ftp.gnome.org/pub/gnome/sources/glib/2.64/glib-2.64.2.tar.xz > glib-2.64.2.tar.xz && \
    tar -xf glib-2.64.2.tar.xz && \
    cd glib-2.64.2 && \
    mkdir ./_build && \
    cd ./_build && \
    meson --prefix=/opt .. && \
    ninja && \
    ninja install

# Install libvips.
#
RUN curl -L https://github.com/libvips/libvips/releases/download/v${VIPS_VERSION}/vips-${VIPS_VERSION}.tar.gz > vips-${VIPS_VERSION}.tar.gz && \
    tar -xf vips-${VIPS_VERSION}.tar.gz && \
    cd vips-${VIPS_VERSION} && \
    ./configure \
      --prefix=/opt \
      --disable-gtk-doc \
      --without-magick \
      --with-expat=/opt \
      --with-giflib-includes=/opt/local/include \
      --with-giflib-libraries=/opt/local/lib && \
    make && \
    make install && \
    echo /opt/lib > /etc/ld.so.conf.d/libvips.conf && \
    ldconfig

RUN cp -a /opt/lib/libvips.so* /build/share/lib && \
    cp -a /opt/lib/libvips-cpp* /build/share/lib

# Copy all pkgconfig, includes for node sharp build.
#
RUN cp -a /opt/lib/pkgconfig /build/share/lib && \
    cp -a /opt/lib64/pkgconfig/* /build/share/lib/pkgconfig && \
    cp -a /opt/include /build/share && \
    cp -a /opt/lib64/glib-2.0 /build/share/lib

# Store the VIPS_VERSION variable in a file, accessible to the deploy script.
#
RUN echo $VIPS_VERSION > "./share/VIPS_VERSION"

# Create an /build/share/opt/lib64 symlink for shared objects.
#
RUN cd ./share && ln -s lib lib64

# Zip up contents so final `lib` can be placed in /opt layer.
#
RUN zip --symlinks -r libvips.zip .

# Zip up contents so final `lib` can be placed in /opt layer.
#
# use lambda nodejs image for amazon linux
FROM public.ecr.aws/lambda/nodejs:16 as release

ENV PATH=/opt/bin:$PATH
ENV LD_LIBRARY_PATH=/opt/lib:/opt/lib64:$LD_LIBRARY_PATH
ENV PKG_CONFIG_PATH=/opt/lib/pkgconfig:/opt/lib64/pkgconfig
ENV CFLAGS="-fexceptions -Wall -O3"
ENV CXXFLAGS="${CFLAGS}"

COPY --from=build /build/share/lib /opt/lib
COPY --from=build /build/share/lib64 /opt/lib64

RUN yum install -y zip bash build-essential

RUN export LD_LIBRARY_PATH=/opt/lib:/opt/lib64:$LD_LIBRARY_PATH
WORKDIR /sharplayer
COPY package.json package.json
RUN export SHARP_IGNORE_GLOBAL_LIBVIPS=true \
  && npm install

RUN yum install -y pkgconfig
RUN ldd node_modules/sharp/build/Release/sharp-linux-x64.node

RUN zip -r sharplayer.zip node_modules package.json

ENTRYPOINT ["tail", "-f", "/dev/null"]```

package.json used in Dockerfile

```{
  "name": "sharplayer",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "config": {
    "libvips": "8.10.2"
  },
  "dependencies": {
    "sharp": "^0.31.2"
  }
}```

### Please provide sample image(s) that help explain this question

<!-- Please provide links to one or more images here. -->

lovell commented 2 years ago

SHARP_IGNORE_GLOBAL_LIBVIPS=true

Did you intend to include this in your script?

I don't mind updating versions to latest

I'd highly recommend you update all your dependencies to latest as there have been many bug fixes.

plangley-champ commented 2 years ago

I appreciate the feedback. I've updated all my dependencies and removed SHARP_IGNORE_GLOBAL_LIBVIPS=true. When I log into release container I was expecting versions to change but still looks to be same. I'm going to admit it, I'm new to packaging c++ libs with node module!! Any advice on what I could be missing. My primary goal is to package sharp/libvips with poppler for pdf support via AWS Lambda. If you have example of that I'll build it locally and let me know what the output should be after building it I would appreciate it.

updated Dockerfile

FROM lambci/lambda:build-ruby2.7 as build

WORKDIR /build

ARG VIPS_VERSION=8.13.3

ENV VIPS_VERSION=$VIPS_VERSION
ENV PATH=/opt/bin:$PATH
ENV LD_LIBRARY_PATH=/opt/lib:/opt/lib64:$LD_LIBRARY_PATH
ENV PKG_CONFIG_PATH=/opt/lib/pkgconfig:/opt/lib64/pkgconfig
ENV CFLAGS="-fexceptions -Wall -O3"
ENV CXXFLAGS="${CFLAGS}"

# Setup Some Dirs
#
RUN mkdir -p \
    share/lib \
    share/include

RUN yum remove -y cmake
RUN yum install -y wget openssl
RUN wget https://cmake.org/files/v3.11/cmake-3.11.0.tar.gz
RUN tar -xvzf cmake-3.11.0.tar.gz;cd cmake-3.11.0;./bootstrap && make && make install

# Install expat
#
RUN curl https://codeload.github.com/libexpat/libexpat/zip/R_2_2_9 > libexpat-R_2_2_9.zip && \
    unzip libexpat-R_2_2_9.zip && \
    cd ./libexpat-R_2_2_9/expat && \
    ./buildconf.sh && \
    ./configure --prefix=/opt && \
    make install

RUN cp -a /opt/lib/libexpat.so* /build/share/lib

# Install libpng
#
RUN curl -L https://fossies.org/linux/misc/libpng-1.6.39.tar.xz > libpng-1.6.39.tar.gz && \
     tar -xf libpng-1.6.39.tar.gz && \
     cd libpng-1.6.39 && \
     ./configure --prefix=/opt --disable-static && \
     make && \
     make install

RUN cp -a /opt/lib/libpng.so* /build/share/lib && \
    cp -a /opt/lib/libpng16.so* /build/share/lib

# Install giflib
#
RUN curl -L https://fossies.org/linux/misc/giflib-5.2.1.tar.gz > giflib-5.2.1.tar.gz && \
    tar -xf giflib-5.2.1.tar.gz && \
    cd giflib-5.2.1 && \
    make && \
    make PREFIX=/opt install

RUN cp -a /opt/lib/libgif.so* /build/share/lib

# Install libjpeg-turbo
#
RUN curl -L https://fossies.org/linux/misc/libjpeg-turbo-2.1.4.tar.gz > libjpeg-turbo-2.1.4.tar.gz && \
    tar -xf libjpeg-turbo-2.1.4.tar.gz && \
    cd libjpeg-turbo-2.1.4 && \
    cmake -DCMAKE_INSTALL_PREFIX=/opt && \
    make && \
    make install

RUN cp -a /opt/lib64/libjpeg.so* /build/share/lib && \
    cp -a /opt/lib64/libturbojpeg.so* /build/share/lib

# Install libimagequant
#
RUN git clone https://github.com/ImageOptim/libimagequant.git && \
    cd ./libimagequant && \
    git checkout 2.12.6 && \
    ./configure --prefix=/opt && \
    make libimagequant.so && \
    make install && \
    echo /opt/lib > /etc/ld.so.conf.d/libimagequant.conf && \
    ldconfig

RUN cp -a /opt/lib/libimagequant.so* /build/share/lib/

# Install libfftw
#
RUN curl -L http://www.fftw.org/fftw-3.3.10.tar.gz > fftw-3.3.10.tar.gz && \
    tar -xf fftw-3.3.10.tar.gz && \
    cd ./fftw-3.3.10 && \
    ./configure \
      --prefix=/opt \
      --enable-shared \
      --disable-static \
      --enable-threads \
      --enable-sse2 \
      --enable-avx && \
    make && \
    make install

RUN cp -a /opt/lib/libfftw3* /build/share/lib/

# Install liborc (perf)
#
RUN curl -Lk https://gstreamer.freedesktop.org/data/src/orc/orc-0.4.26.tar.xz > orc-0.4.26.tar.xz && \
    tar -xf orc-0.4.26.tar.xz && \
    cd orc-0.4.26 && \
    ./configure --prefix=/opt && \
    make && \
    make install
RUN cp -a /opt/lib/liborc-0.4.so* /build/share/lib/

# Install libvips. Primary deps https://libvips.github.io/libvips/install.html
#
RUN yum install -y \
    gtk-doc \
    ninja-build \
    wget \
    libffi-devel \
    gobject-introspectioni

RUN pip3 install meson && \
    pip3 install ninja

RUN cp -a /usr/lib64/libgio-2.0* /build/share/lib && \
    cp -a /usr/lib64/libglib-2.0.so* /build/share/lib && \
    cp -a /usr/lib64/libgmodule-2.0.so* /build/share/lib && \
    cp -a /usr/lib64/libgobject-2.0.so* /build/share/lib && \
    cp -a /usr/lib64/libgthread-2.0.so* /build/share/lib && \ 
    cp -a /usr/lib64/libffi.so* /build/share/lib

RUN yum install -y pcre-devel
RUN curl -Lk http://ftp.gnome.org/pub/gnome/sources/glib/2.74/glib-2.74.3.tar.xz > glib-2.74.3.tar.xz && \
    tar -xf glib-2.74.3.tar.xz && \
    cd glib-2.74.3 && \
    mkdir ./_build && \
    cd ./_build && \
    meson --prefix=/opt .. && \
    ninja && \
    ninja install

# Install libvips.
#
RUN curl -L https://github.com/libvips/libvips/releases/download/v${VIPS_VERSION}/vips-${VIPS_VERSION}.tar.gz > vips-${VIPS_VERSION}.tar.gz && \
    tar -xf vips-${VIPS_VERSION}.tar.gz && \
    cd vips-${VIPS_VERSION} && \
    ./configure \
      --prefix=/opt \
      --disable-gtk-doc \
      --without-magick \
      --with-expat=/opt \
      --with-giflib-includes=/opt/local/include \
      --with-giflib-libraries=/opt/local/lib && \
    make && \
    make install && \
    echo /opt/lib > /etc/ld.so.conf.d/libvips.conf && \
    ldconfig

RUN cp -a /opt/lib/libvips.so* /build/share/lib && \
    cp -a /opt/lib/libvips-cpp* /build/share/lib

# Copy all pkgconfig, includes for node sharp build.
#
RUN cp -a /opt/lib/pkgconfig /build/share/lib && \
    cp -a /opt/lib64/pkgconfig/* /build/share/lib/pkgconfig && \
    cp -a /opt/include /build/share && \
    cp -a /opt/lib64/glib-2.0 /build/share/lib

# Store the VIPS_VERSION variable in a file, accessible to the deploy script.
#
RUN echo $VIPS_VERSION > "./share/VIPS_VERSION"

# Create an /build/share/opt/lib64 symlink for shared objects.
#
RUN cd ./share && ln -s lib lib64

# Zip up contents so final `lib` can be placed in /opt layer.
#
RUN zip --symlinks -r libvips.zip .

# Zip up contents so final `lib` can be placed in /opt layer.
#
# use lambda nodejs image for amazon linux
FROM public.ecr.aws/lambda/nodejs:16 as release

ENV PATH=/opt/bin:$PATH
ENV LD_LIBRARY_PATH=/opt/lib:/opt/lib64:$LD_LIBRARY_PATH
ENV PKG_CONFIG_PATH=/opt/lib/pkgconfig:/opt/lib64/pkgconfig
ENV CFLAGS="-fexceptions -Wall -O3"
ENV CXXFLAGS="${CFLAGS}"

COPY --from=build /build/share/lib /opt/lib
COPY --from=build /build/share/lib64 /opt/lib64

RUN yum install -y zip bash build-essential

RUN export LD_LIBRARY_PATH=/opt/lib:/opt/lib64:$LD_LIBRARY_PATH
WORKDIR /sharplayer
COPY package.json package.json
RUN npm install

RUN yum install -y pkgconfig
RUN ldd node_modules/sharp/build/Release/sharp-linux-x64.node

RUN zip -r sharplayer.zip node_modules package.json

plangley-champ commented 2 years ago

versions output

{
  "aom": "3.5.0",
  "cairo": "1.17.6",
  "cgif": "0.3.0",
  "exif": "0.6.24",
  "expat": "2.5.0",
  "ffi": "3.4.4",
  "fontconfig": "2.14.1",
  "freetype": "2.12.1",
  "fribidi": "1.0.12",
  "gdkpixbuf": "2.42.10",
  "glib": "2.74.1",
  "gsf": "1.14.50",
  "harfbuzz": "5.3.1",
  "heif": "1.13.0",
  "imagequant": "2.4.1",
  "lcms": "2.14",
  "mozjpeg": "4.1.1",
  "orc": "0.4.33",
  "pango": "1.50.11",
  "pixman": "0.42.2",
  "png": "1.6.38",
  "proxy-libintl": "0.4",
  "svg": "2.55.1",
  "spng": "0.7.2",
  "tiff": "4.4.0",
  "vips": "8.13.3",
  "webp": "1.2.4",
  "xml": "2.10.3",
  "zlib-ng": "2.0.6"
}

plangley-champ commented 1 year ago

I guess that's my question. If I have custom version added or update lib in set will version change in my versions output above?

plangley-champ commented 1 year ago

example, "glib": "2.74.3", is installed, but output shows "glib": "2.74.1",

lovell commented 1 year ago

If you're using a custom libvips at runtime, sharp.versions will contain an entry only for vips.

The output of RUN npm install --verbose --foreground-scripts might provide some information. Perhaps not all the pkg-config files are being copied?

I can't vouch for any of them, but there are a few repos on GitHub where others maintain custom Lambda layers that you may be able to borrow ideas from - see https://github.com/search?q=sharp+lambda+layer&type=repositories

atxplangl commented 1 year ago

@lovell you can close issue and appreciate your feedback. I was able to deploy sharp + custom libvips into lambda with no issues. If others in future have similar issues I found the following the best way to validate if you're using custom libvips w/ sharp

test locally in docker
execute ldd --verbose /path/to/node_modules/sharp/build/Release/sharp-linux-x64.node | grep ldd returns custom lib output then you're probably good

Kikobeats commented 1 year ago

I'm still experiencing issues, and I make it easy to reproduce in https://github.com/microlinkhq/splashy repo.

Here is the thing:

sharp v0.31.1 works fine: https://github.com/microlinkhq/splashy/pull/30
sharp v0.31.2 failed: https://github.com/microlinkhq/splashy/pull/36
sharp v0.31.3 continue failing: https://github.com/microlinkhq/splashy/pull/39

lovell commented 1 year ago

@Kikobeats Please open a new installation issue and answer all of the questions if you still require help.

Kikobeats commented 1 year ago

@lovell makes sense; done at https://github.com/lovell/sharp/issues/3504 🙂 I tried to attach all the details possible

lovell / sharp