Closed aslafy-z closed 1 year ago
Despite changes above I've provided, this only allows for getting libvips
(vips-dev) to build. The npm sharp
package will also build using it, but attempting to run the actual container with gatsby develop
spouts a tonne of errors to do with the gatsby-plugin-sharp
afaik. Building that seems to cause errors too..
This appears to be due to using the edge repositories for libvips
. They're stated to not be reliable when used with non-edge versions of Alpine.. That'd explain why a similar Docker Image was building fine in the past but now also fails to build. The npm package sharp
will install binaries if it doesn't detect a local libvips
install on Alpine 3.8+ for nodejs v8 or v10.
I've recently got a Docker Image that actually builds and runs properly. I had success with this Dockerfile and removing the libvips
dependency install, but for some reason couldn't connect over localhost:8000
to the running gatsby develop
server. Not wanting to spend more time debugging why that is, I used a bit of the other linked Docker Image files which fixed that issue.
Dockerfile
FROM node:10-alpine
EXPOSE 8000
ARG GATSBY_VERSION=^2.0.0
# Install dev dependencies
RUN set -x \
&& apk add --no-cache \
g++ \
git \
make \
python2 \
binutils
# Install gatsby cli and cleanup
RUN set -x \
&& npm install --global gatsby@${GATSBY_VERSION} \
--no-optional gatsby@${GATSBY_VERSION} \
&& npm cache clean --force
RUN mkdir -p /site
WORKDIR /site
VOLUME /site
COPY ./entry.sh /
RUN chmod +x /entry.sh
ENTRYPOINT ["/entry.sh"]
entry.sh
#!/bin/sh
set -e
export GATSBY_DIR="/site"
export PATH="$PATH:/usr/local/bin/gatsby"
# Initialize Gatsby or run NPM install if needed
if [ ! -f "$GATSBY_DIR/package.json" ]
then
echo "Initializing Gatsby..."
gatsby new $GATSBY_DIR
else
if [ ! -e "$GATSBY_DIR/node_modules/" ]
then
echo "Node modules is empty. Running npm install..."
npm install
fi
fi
# Decide what to do
if [ "$1" == "develop" ]
then
rm -rf $GATSBY_DIR/public
gatsby develop --host 0.0.0.0
elif [ "$1" == "build" ]
then
rm -rf $GATSBY_DIR/public
gatsby build
elif [ "$1" == "stage" ]
then
rm -rf $GATSBY_DIR/public
gatsby build
gatsby serve --port 8000
else
exec $@
fi
Build: docker build -t docker-gatsby .
Run: docker run -it --rm -v $(pwd)/site:/site -p 8000:8000 docker-gatsby develop
Maybe if I find some time to work on it a bit more I could make another PR. Just thought I'd share this as if you try building the current PR and using it now, I think you'll run into the problems I've been pointing out.
I applied all the suggestions and dropped the alpine libvips
dependency, fixed cli handling also. I was waiting for the sharp's alpine support to update this PR. Thanks to all the reviewers!
@cusspvz Let me know about the name convention!
I did come across another issue many times when messing around with getting a working Docker image, I believe it also happened with the above config I shared here. When running something like gatsby develop
it worked fine, but then I tried with my prior existing project package.json and got issues like this:
error Plugin gatsby-plugin-manifest returned an error
Error: Could not locate the bindings file. Tried:
→ /site/node_modules/sharp/build/sharp.node
→ /site/node_modules/sharp/build/Debug/sharp.node
→ /site/node_modules/sharp/build/Release/sharp.node
→ /site/node_modules/sharp/out/Debug/sharp.node
→ /site/node_modules/sharp/Debug/sharp.node
→ /site/node_modules/sharp/out/Release/sharp.node
→ /site/node_modules/sharp/Release/sharp.node
→ /site/node_modules/sharp/build/default/sharp.node
→ /site/node_modules/sharp/compiled/10.14.2/linux/x64/sharp.node
- bindings.js:96 bindings
[site]/[bindings]/bindings.js:96:9
- constructor.js:10 Object.<anonymous>
[site]/[sharp]/lib/constructor.js:10:34
- v8-compile-cache.js:178 Module._compile
[site]/[v8-compile-cache]/v8-compile-cache.js:178:30
- loader.js:700 Object.Module._extensions..js
internal/modules/cjs/loader.js:700:10
- loader.js:599 Module.load
internal/modules/cjs/loader.js:599:32
- loader.js:538 tryModuleLoad
internal/modules/cjs/loader.js:538:12
- loader.js:530 Function.Module._load
internal/modules/cjs/loader.js:530:3
- loader.js:637 Module.require
internal/modules/cjs/loader.js:637:17
- v8-compile-cache.js:159 require
[site]/[v8-compile-cache]/v8-compile-cache.js:159:20
- index.js:3 Object.<anonymous>
[site]/[sharp]/lib/index.js:3:15
- v8-compile-cache.js:178 Module._compile
[site]/[v8-compile-cache]/v8-compile-cache.js:178:30
- loader.js:700 Object.Module._extensions..js
internal/modules/cjs/loader.js:700:10
- loader.js:599 Module.load
internal/modules/cjs/loader.js:599:32
- loader.js:538 tryModuleLoad
internal/modules/cjs/loader.js:538:12
- loader.js:530 Function.Module._load
internal/modules/cjs/loader.js:530:3
- loader.js:637 Module.require
internal/modules/cjs/loader.js:637:17
- v8-compile-cache.js:159 require
[site]/[v8-compile-cache]/v8-compile-cache.js:159:20
- gatsby-node.js:13 Object.<anonymous>
[site]/[gatsby-plugin-manifest]/gatsby-node.js:13:13
- v8-compile-cache.js:178 Module._compile
[site]/[v8-compile-cache]/v8-compile-cache.js:178:30
- loader.js:700 Object.Module._extensions..js
internal/modules/cjs/loader.js:700:10
- loader.js:599 Module.load
internal/modules/cjs/loader.js:599:32
- loader.js:538 tryModuleLoad
internal/modules/cjs/loader.js:538:12
It was resolved by adding bindings.js. But that's something that wouldn't belong here. For some reason, even if the .node file for sharp was where it said it looked, it wasn't able to find it and threw those errors.
TL;DR: gatsby-plugin-sharp
has some errors when converting png or jpeg to webp, there's also errors for jpeg if useMozjpeg
is enabled. See Dockerfile at end for additional dependencies and comments. Rest is just sharing the error logs and tweaking dependencies/size.
I have been playing around with my docker image variant again. Mostly because I wanted to address the crashes I was getting with gatsby-plugin-sharp
that the installed default starter users was causing.
2 kinds of errors. One was if you used the _withWebp
fragments in GraphQL queries for images, the binary cwebp would panic as we're missing jpeg/png support for the conversion to webp.
Error: Command failed: /site/node_modules/cwebp-bin/vendor/cwebp -quiet -mt -q 50 -o /tmp/a740d783-1a1e-40ba-a2be-e9ae8c376b6a /tmp/364 7a55e-1476-44b2-ad2d-e70fec58bcde
JPEG support not compiled. Please install the libjpeg development package before building.
Error! Could not process file /tmp/3647a55e-1476-44b2-ad2d-e70fec58bcde
Error! Cannot read input picture file '/tmp/3647a55e-1476-44b2-ad2d-e70fec58bcde'
Error: Command failed: /site/node_modules/cwebp-bin/vendor/cwebp -quiet -mt -q 50 -o /tmp/d3d5afd6-32cd-497f-81c1-db1da83455b8 /tmp/bc1 f28cd-a14a-46e1-967a-66f04a723cf2
PNG support not compiled. Please install the libpng development package before building.
Error! Could not process file /tmp/bc1f28cd-a14a-46e1-967a-66f04a723cf2
Error! Cannot read input picture file '/tmp/bc1f28cd-a14a-46e1-967a-66f04a723cf2'
Also:
⚠ Command failed: /site/node_modules/pngquant-bin/vendor/pngquant --version
⚠ pngquant pre-build test failed
ℹ compiling from source
✖ Error: pngquant failed to build, make sure that libpng-dev is installed
Solved by adding lcms2-dev
and libpng-dev
.
The other issue was only triggered if you enabled mozjpeg
in gatsby-config.js
for gatsby-plugin-sharp
explicitly. When converting to JPG if it's not available(it builds but raises a install error without failing npm install) you'll get a crash like this:
Error: spawn /site/node_modules/mozjpeg/vendor/cjpeg ENOENT
The npm install failed because of this:
> mozjpeg@6.0.1 postinstall /site/node_modules/mozjpeg
> node lib/install.js
⚠ spawn /site/node_modules/mozjpeg/vendor/cjpeg ENOENT
⚠ mozjpeg pre-build test failed
ℹ compiling from source
✖ Error: Command failed: /bin/sh -c autoreconf -fiv
/bin/sh: autoreconf: not found
# Or `autoreconf` is found from `autoconf`:
> mozjpeg@6.0.1 postinstall /site/node_modules/mozjpeg
> node lib/install.js
⚠ spawn /site/node_modules/mozjpeg/vendor/cjpeg ENOENT
⚠ mozjpeg pre-build test failed
ℹ compiling from source
✖ Error: Command failed: /bin/sh -c autoreconf -fiv
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: running: /usr/bin/autoconf --force
configure.ac:23: error: possibly undefined macro: AC_PROG_LIBTOOL
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
autoreconf: /usr/bin/autoconf failed with exit status: 1
So you can get that with autoconf
but that'll fail without automake
. EDIT: AFAIK it worked fine in the past, might have been because I removed other dependencies I had earlier, also needs libtool
and nasm
to build.
I've built the image multiple times and cleared out the mounted volume to reinstall again many times. Quite often the npm install appears to go fine sometimes yet if you add a package later or npm install
(I often clear out ./node_modules
again and run that to test), you get build/install issues like I've shown here.
I had removed python2
(40MB in weight) and not had any problems since, I know node-gyp relies on it but wasn't sure if it was needed for the image and if it a user needed it, they could extend the docker image to include it... just ran into a failure though while preparing this issue. I think the problem was due to connection dropping which prevented getting the full file, so still on the fence if we need it?
g++
brings almost 150MB of weight with it. I thought without vips-dev
/libvips we could drop it. However the plugins mentioned above need to compile from source unless there is a way for us to provide them prior to the install? Without these two dependencies that's a fairly good weight drop of close to 200MB. The 46 new dependencies add 40MB50MB(bulk of it is for mozjpeg).
One other optimization that I think is relevant to make here, gatsby
is installed globally, but do you need the full package? It's about 150MB in weight, meanwhile, gatsby-cli
is only 15MB! :)
These both want a c compiler, pngquant does too I think, though it's error just states it failed and if I made sure to have libpng-dev
. Fixed with nvm, g++ is neededgcc
(85MB):
> sharp@0.21.1 install /site/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linuxmusl-x64.tar.gz
> cwebp-bin@5.0.0 postinstall /site/node_modules/cwebp-bin
> node lib/install.js
⚠ spawn /site/node_modules/cwebp-bin/vendor/cwebp ENOENT
⚠ cwebp pre-build test failed
ℹ compiling from source
✖ Error: Command failed: /bin/sh -c ./configure --disable-shared --prefix="/site/node_modules/cwebp-bin/vendor" --bindir="/site/node_modules/cwebp-bin/vendor"
configure: error: in `/site/node_modules/cwebp-bin/1da73eca-46cc-4eee-afa2-65f12f6b5f5c':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for style of include used by make... GNU
checking for gcc... no
checking for cc... no
checking for cl.exe... no
at Promise.all.then.arr (/site/node_modules/bin-build/node_modules/execa/index.js:231:11)
at process._tickCallback (internal/process/next_tick.js:68:7)
> mozjpeg@6.0.1 postinstall /site/node_modules/mozjpeg
> node lib/install.js
⚠ spawn /site/node_modules/mozjpeg/vendor/cjpeg ENOENT
⚠ mozjpeg pre-build test failed
ℹ compiling from source
✖ Error: Command failed: /bin/sh -c ./configure --enable-static --disable-shared --disable-dependency-tracking --with-jpeg8 --prefix="/site/node_modules/mozjpeg/vendor" --bindir="/site/node_modules/mozjpeg/vendor" --libdir="/site/node_modules/mozjpeg/vendor"
configure: error: in `/site/node_modules/mozjpeg/16a8f62e-2108-4f5f-8251-cce60eaba872':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking whether make supports the include directive... yes (GNU style)
checking for gcc... no
checking for cc... no
checking for cl.exe... no
Output from trying gcc instead of g++ to compile c:
> sharp@0.21.1 install /site/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)
info sharp Using cached /root/.npm/_libvips/libvips-8.7.0-linuxmusl-x64.tar.gz
> cwebp-bin@5.0.0 postinstall /site/node_modules/cwebp-bin
> node lib/install.js
⚠ spawn /site/node_modules/cwebp-bin/vendor/cwebp ENOENT
⚠ cwebp pre-build test failed
ℹ compiling from source
✖ Error: Command failed: /bin/sh -c ./configure --disable-shared --prefix="/site/node_modules/cwebp-bin/vendor" --bindir="/site/node_modules/cwebp-bin/vendor"
configure: error: in `/site/node_modules/cwebp-bin/032cf998-75a9-46aa-8709-a32909a5a825':
configure: error: C compiler cannot create executables
See `config.log' for more details
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... no
at Promise.all.then.arr (/site/node_modules/bin-build/node_modules/execa/index.js:231:11)
at process._tickCallback (internal/process/next_tick.js:68:7)
> mozjpeg@6.0.1 postinstall /site/node_modules/mozjpeg
> node lib/install.js
⚠ spawn /site/node_modules/mozjpeg/vendor/cjpeg ENOENT
⚠ mozjpeg pre-build test failed
ℹ compiling from source
✖ Error: Command failed: /bin/sh -c ./configure --enable-static --disable-shared --disable-dependency-tracking --with-jpeg8 --prefix="/site/node_modules/mozjpeg/vendor" --bindir="/site/node_modules/mozjpeg/vendor" --libdir="/site/node_modules/mozjpeg/vendor"
configure: error: in `/site/node_modules/mozjpeg/18f42ca2-c988-4d9c-9b92-ce6b9a42f50e':
configure: error: C compiler cannot create executables
See `config.log' for more details
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking whether make supports the include directive... yes (GNU style)
checking for gcc... gcc
checking whether the C compiler works... no
Full Dockerfile:
# Use gatsby-cli instead of gatsby, roughly 150MB vs 15MB difference!
# Remove dev dependencies that aren't required anymore(not building vips-dev),
# if a user needs to build for specific package they should build a new docker image extending this one
# python2 is about 40MB and might be useful for packages needing node-gyp to build
# g++ is about 150MB, gcc 85MB
FROM node:10-alpine # 70MB
# Install dev dependencies (gcc 85MB, git 15MB)
RUN set -x \
&& apk add --no-cache \
gcc \
git \
make
# Install gatsby cli and cleanup (adds about 15MB)
ARG GATSBY_VERSION=^2.0.0
RUN set -x \
&& npm install --global gatsby-cli@${GATSBY_VERSION} \
--no-optional gatsby-cli@${GATSBY_VERSION} \
&& npm cache clean --force
# These let toFormat work for image format conversions with webp
# imagemin plugins support(part of gatsby-plugin-sharp) (adds about 5MB)
# lcms2-dev - jpeg and tiff support
# libpng-dev - png support
RUN set -x \
&& apk add --no-cache \
lcms2-dev \
libpng-dev
# mozjpeg support 46MB (also depends on libpng-dev)
RUN set -x \
&& apk add --no-cache \
autoconf \
automake \
libtool \
nasm
RUN mkdir -p /site
WORKDIR /site
VOLUME /site
EXPOSE 8000
COPY ./entry.sh /
RUN chmod +x /entry.sh
ENTRYPOINT ["/entry.sh"]
EDIT:
While looking into multi-stage build for mozjpeg, it seems that they've recently switched away from autoreconf and use cmake now, so the current npm package is a tad out of date. Thus the dependencies aren't so relevant anymore..
Closes #16 #11
node:10-alpine
Build with:
Works best with a volume:
User is specified to ensure it doesn't fucked up your files permission.
Supported commands:
develop
->gatsby develop
build
->gatsby build
serve
->gatsby serve
stage
->gatsby build && gatsby serve
These commands need to have wrappers because they don't support configuration of host and port via variables environments. Too bad :(
You can also use others commands of
gatsby
which does not need a wrapper (eg.:repl
,new
,info
).If a folder without
package.json
file is mounted on the/site
dir, it will be provisioned with https://github.com/gatsbyjs/gatsby-starter-default.