gatsbyjs / gatsby-docker

Docker image that builds and hosts a Gatsby site
MIT License
356 stars 88 forks source link

Add development image #19

Closed aslafy-z closed 1 year ago

aslafy-z commented 5 years ago

Closes #16 #11

Build with:

docker build -t gatsbyjs/cli -f Dockerfile.cli .

Works best with a volume:

mkdir -p gatsby-website
docker run -it --rm -p 8000:8000 \
    --user "$(id -u):$(id -g)" \
    -v $(pwd)/gatsby-website:/site \
    gatsbyjs/cli develop

User is specified to ensure it doesn't fucked up your files permission.

Supported commands:

These commands need to have wrappers because they don't support configuration of host and port via variables environments. Too bad :(

You can also use others commands of gatsby which does not need a wrapper (eg.: repl, new, info).

If a folder without package.json file is mounted on the /site dir, it will be provisioned with https://github.com/gatsbyjs/gatsby-starter-default.

polarathene commented 5 years ago

Despite changes above I've provided, this only allows for getting libvips(vips-dev) to build. The npm sharp package will also build using it, but attempting to run the actual container with gatsby develop spouts a tonne of errors to do with the gatsby-plugin-sharp afaik. Building that seems to cause errors too..

This appears to be due to using the edge repositories for libvips. They're stated to not be reliable when used with non-edge versions of Alpine.. That'd explain why a similar Docker Image was building fine in the past but now also fails to build. The npm package sharp will install binaries if it doesn't detect a local libvips install on Alpine 3.8+ for nodejs v8 or v10.

I've recently got a Docker Image that actually builds and runs properly. I had success with this Dockerfile and removing the libvips dependency install, but for some reason couldn't connect over localhost:8000 to the running gatsby develop server. Not wanting to spend more time debugging why that is, I used a bit of the other linked Docker Image files which fixed that issue.

Dockerfile

FROM node:10-alpine

EXPOSE 8000

ARG GATSBY_VERSION=^2.0.0

# Install dev dependencies
RUN set -x \
  && apk add --no-cache \
  g++ \
  git \
  make \
  python2 \
  binutils

# Install gatsby cli and cleanup
RUN set -x \
  && npm install --global gatsby@${GATSBY_VERSION} \
  --no-optional gatsby@${GATSBY_VERSION} \
  && npm cache clean --force

RUN mkdir -p /site
WORKDIR /site
VOLUME /site

COPY ./entry.sh /
RUN chmod +x /entry.sh
ENTRYPOINT ["/entry.sh"]

entry.sh

#!/bin/sh
set -e

export GATSBY_DIR="/site"
export PATH="$PATH:/usr/local/bin/gatsby"

# Initialize Gatsby or run NPM install if needed
if [ ! -f "$GATSBY_DIR/package.json" ]
then
  echo "Initializing Gatsby..."
  gatsby new $GATSBY_DIR

else
  if [ ! -e "$GATSBY_DIR/node_modules/" ]
  then
    echo "Node modules is empty. Running npm install..."
    npm install

  fi

fi

# Decide what to do
if  [ "$1" == "develop" ]
then
  rm -rf $GATSBY_DIR/public
  gatsby develop --host 0.0.0.0

elif  [ "$1" == "build" ]
then
  rm -rf $GATSBY_DIR/public
  gatsby build

elif  [ "$1" == "stage" ]
then
  rm -rf $GATSBY_DIR/public
  gatsby build
  gatsby serve --port 8000

else
  exec $@

fi

Build: docker build -t docker-gatsby . Run: docker run -it --rm -v $(pwd)/site:/site -p 8000:8000 docker-gatsby develop

Maybe if I find some time to work on it a bit more I could make another PR. Just thought I'd share this as if you try building the current PR and using it now, I think you'll run into the problems I've been pointing out.

aslafy-z commented 5 years ago

I applied all the suggestions and dropped the alpine libvips dependency, fixed cli handling also. I was waiting for the sharp's alpine support to update this PR. Thanks to all the reviewers!

@cusspvz Let me know about the name convention!

polarathene commented 5 years ago

I did come across another issue many times when messing around with getting a working Docker image, I believe it also happened with the above config I shared here. When running something like gatsby develop it worked fine, but then I tried with my prior existing project package.json and got issues like this:

error Plugin gatsby-plugin-manifest returned an error

  Error: Could not locate the bindings file. Tried:
   → /site/node_modules/sharp/build/sharp.node
   → /site/node_modules/sharp/build/Debug/sharp.node
   → /site/node_modules/sharp/build/Release/sharp.node
   → /site/node_modules/sharp/out/Debug/sharp.node
   → /site/node_modules/sharp/Debug/sharp.node
   → /site/node_modules/sharp/out/Release/sharp.node
   → /site/node_modules/sharp/Release/sharp.node
   → /site/node_modules/sharp/build/default/sharp.node
   → /site/node_modules/sharp/compiled/10.14.2/linux/x64/sharp.node

  - bindings.js:96 bindings
    [site]/[bindings]/bindings.js:96:9

  - constructor.js:10 Object.<anonymous>
    [site]/[sharp]/lib/constructor.js:10:34

  - v8-compile-cache.js:178 Module._compile
    [site]/[v8-compile-cache]/v8-compile-cache.js:178:30

  - loader.js:700 Object.Module._extensions..js
    internal/modules/cjs/loader.js:700:10

  - loader.js:599 Module.load
    internal/modules/cjs/loader.js:599:32

  - loader.js:538 tryModuleLoad
    internal/modules/cjs/loader.js:538:12

  - loader.js:530 Function.Module._load
    internal/modules/cjs/loader.js:530:3

  - loader.js:637 Module.require
    internal/modules/cjs/loader.js:637:17

  - v8-compile-cache.js:159 require
    [site]/[v8-compile-cache]/v8-compile-cache.js:159:20

  - index.js:3 Object.<anonymous>
    [site]/[sharp]/lib/index.js:3:15

  - v8-compile-cache.js:178 Module._compile
    [site]/[v8-compile-cache]/v8-compile-cache.js:178:30

  - loader.js:700 Object.Module._extensions..js
    internal/modules/cjs/loader.js:700:10

  - loader.js:599 Module.load
    internal/modules/cjs/loader.js:599:32

  - loader.js:538 tryModuleLoad
    internal/modules/cjs/loader.js:538:12

  - loader.js:530 Function.Module._load
    internal/modules/cjs/loader.js:530:3

  - loader.js:637 Module.require
    internal/modules/cjs/loader.js:637:17

  - v8-compile-cache.js:159 require
    [site]/[v8-compile-cache]/v8-compile-cache.js:159:20

  - gatsby-node.js:13 Object.<anonymous>
    [site]/[gatsby-plugin-manifest]/gatsby-node.js:13:13

  - v8-compile-cache.js:178 Module._compile
    [site]/[v8-compile-cache]/v8-compile-cache.js:178:30

  - loader.js:700 Object.Module._extensions..js
    internal/modules/cjs/loader.js:700:10

  - loader.js:599 Module.load
    internal/modules/cjs/loader.js:599:32

  - loader.js:538 tryModuleLoad
    internal/modules/cjs/loader.js:538:12

It was resolved by adding bindings.js. But that's something that wouldn't belong here. For some reason, even if the .node file for sharp was where it said it looked, it wasn't able to find it and threw those errors.

polarathene commented 5 years ago

TL;DR: gatsby-plugin-sharp has some errors when converting png or jpeg to webp, there's also errors for jpeg if useMozjpeg is enabled. See Dockerfile at end for additional dependencies and comments. Rest is just sharing the error logs and tweaking dependencies/size.


I have been playing around with my docker image variant again. Mostly because I wanted to address the crashes I was getting with gatsby-plugin-sharp that the installed default starter users was causing.

2 kinds of errors. One was if you used the _withWebp fragments in GraphQL queries for images, the binary cwebp would panic as we're missing jpeg/png support for the conversion to webp.

  Error: Command failed: /site/node_modules/cwebp-bin/vendor/cwebp -quiet -mt -q 50 -o /tmp/a740d783-1a1e-40ba-a2be-e9ae8c376b6a /tmp/364  7a55e-1476-44b2-ad2d-e70fec58bcde
  JPEG support not compiled. Please install the libjpeg development package before building.
  Error! Could not process file /tmp/3647a55e-1476-44b2-ad2d-e70fec58bcde
  Error! Cannot read input picture file '/tmp/3647a55e-1476-44b2-ad2d-e70fec58bcde'

  Error: Command failed: /site/node_modules/cwebp-bin/vendor/cwebp -quiet -mt -q 50 -o /tmp/d3d5afd6-32cd-497f-81c1-db1da83455b8 /tmp/bc1  f28cd-a14a-46e1-967a-66f04a723cf2
  PNG support not compiled. Please install the libpng development package before building.
  Error! Could not process file /tmp/bc1f28cd-a14a-46e1-967a-66f04a723cf2
  Error! Cannot read input picture file '/tmp/bc1f28cd-a14a-46e1-967a-66f04a723cf2'

Also:

⚠ Command failed: /site/node_modules/pngquant-bin/vendor/pngquant --version

  ⚠ pngquant pre-build test failed
  ℹ compiling from source
  ✖ Error: pngquant failed to build, make sure that libpng-dev is installed

Solved by adding lcms2-dev and libpng-dev.

The other issue was only triggered if you enabled mozjpeg in gatsby-config.js for gatsby-plugin-sharp explicitly. When converting to JPG if it's not available(it builds but raises a install error without failing npm install) you'll get a crash like this:

  Error: spawn /site/node_modules/mozjpeg/vendor/cjpeg ENOENT

The npm install failed because of this:

> mozjpeg@6.0.1 postinstall /site/node_modules/mozjpeg
> node lib/install.js

  ⚠ spawn /site/node_modules/mozjpeg/vendor/cjpeg ENOENT
  ⚠ mozjpeg pre-build test failed
  ℹ compiling from source
  ✖ Error: Command failed: /bin/sh -c autoreconf -fiv
/bin/sh: autoreconf: not found

# Or `autoreconf` is found from `autoconf`:

> mozjpeg@6.0.1 postinstall /site/node_modules/mozjpeg
> node lib/install.js

  ⚠ spawn /site/node_modules/mozjpeg/vendor/cjpeg ENOENT
  ⚠ mozjpeg pre-build test failed
  ℹ compiling from source
  ✖ Error: Command failed: /bin/sh -c autoreconf -fiv
autoreconf: Entering directory `.'
autoreconf: configure.ac: not using Gettext
autoreconf: running: aclocal --force 
autoreconf: configure.ac: tracing
autoreconf: configure.ac: not using Libtool
autoreconf: running: /usr/bin/autoconf --force
configure.ac:23: error: possibly undefined macro: AC_PROG_LIBTOOL
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
autoreconf: /usr/bin/autoconf failed with exit status: 1

So you can get that with autoconf but that'll fail without automake. EDIT: AFAIK it worked fine in the past, might have been because I removed other dependencies I had earlier, also needs libtool and nasm to build.

I've built the image multiple times and cleared out the mounted volume to reinstall again many times. Quite often the npm install appears to go fine sometimes yet if you add a package later or npm install(I often clear out ./node_modules again and run that to test), you get build/install issues like I've shown here.

I had removed python2(40MB in weight) and not had any problems since, I know node-gyp relies on it but wasn't sure if it was needed for the image and if it a user needed it, they could extend the docker image to include it... just ran into a failure though while preparing this issue. I think the problem was due to connection dropping which prevented getting the full file, so still on the fence if we need it?

g++ brings almost 150MB of weight with it. I thought without vips-dev/libvips we could drop it. However the plugins mentioned above need to compile from source unless there is a way for us to provide them prior to the install? Without these two dependencies that's a fairly good weight drop of close to 200MB. The 46 new dependencies add 40MB50MB(bulk of it is for mozjpeg).

One other optimization that I think is relevant to make here, gatsby is installed globally, but do you need the full package? It's about 150MB in weight, meanwhile, gatsby-cli is only 15MB! :)

These both want a c compiler, pngquant does too I think, though it's error just states it failed and if I made sure to have libpng-dev. Fixed with gcc(85MB): nvm, g++ is needed

> sharp@0.21.1 install /site/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.7.0/libvips-8.7.0-linuxmusl-x64.tar.gz

> cwebp-bin@5.0.0 postinstall /site/node_modules/cwebp-bin
> node lib/install.js

  ⚠ spawn /site/node_modules/cwebp-bin/vendor/cwebp ENOENT
  ⚠ cwebp pre-build test failed
  ℹ compiling from source
  ✖ Error: Command failed: /bin/sh -c ./configure --disable-shared --prefix="/site/node_modules/cwebp-bin/vendor" --bindir="/site/node_modules/cwebp-bin/vendor"
configure: error: in `/site/node_modules/cwebp-bin/1da73eca-46cc-4eee-afa2-65f12f6b5f5c':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details

checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for style of include used by make... GNU
checking for gcc... no
checking for cc... no
checking for cl.exe... no

    at Promise.all.then.arr (/site/node_modules/bin-build/node_modules/execa/index.js:231:11)
    at process._tickCallback (internal/process/next_tick.js:68:7)

> mozjpeg@6.0.1 postinstall /site/node_modules/mozjpeg
> node lib/install.js

  ⚠ spawn /site/node_modules/mozjpeg/vendor/cjpeg ENOENT
  ⚠ mozjpeg pre-build test failed
  ℹ compiling from source
  ✖ Error: Command failed: /bin/sh -c ./configure --enable-static --disable-shared --disable-dependency-tracking --with-jpeg8  --prefix="/site/node_modules/mozjpeg/vendor" --bindir="/site/node_modules/mozjpeg/vendor" --libdir="/site/node_modules/mozjpeg/vendor"
configure: error: in `/site/node_modules/mozjpeg/16a8f62e-2108-4f5f-8251-cce60eaba872':
configure: error: no acceptable C compiler found in $PATH
See `config.log' for more details

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking whether make supports the include directive... yes (GNU style)
checking for gcc... no
checking for cc... no
checking for cl.exe... no

Output from trying gcc instead of g++ to compile c:

> sharp@0.21.1 install /site/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

info sharp Using cached /root/.npm/_libvips/libvips-8.7.0-linuxmusl-x64.tar.gz

> cwebp-bin@5.0.0 postinstall /site/node_modules/cwebp-bin
> node lib/install.js

  ⚠ spawn /site/node_modules/cwebp-bin/vendor/cwebp ENOENT
  ⚠ cwebp pre-build test failed
  ℹ compiling from source
  ✖ Error: Command failed: /bin/sh -c ./configure --disable-shared --prefix="/site/node_modules/cwebp-bin/vendor" --bindir="/site/node_modules/cwebp-bin/vendor"
configure: error: in `/site/node_modules/cwebp-bin/032cf998-75a9-46aa-8709-a32909a5a825':
configure: error: C compiler cannot create executables
See `config.log' for more details

checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... no

    at Promise.all.then.arr (/site/node_modules/bin-build/node_modules/execa/index.js:231:11)
    at process._tickCallback (internal/process/next_tick.js:68:7)

> mozjpeg@6.0.1 postinstall /site/node_modules/mozjpeg
> node lib/install.js

  ⚠ spawn /site/node_modules/mozjpeg/vendor/cjpeg ENOENT
  ⚠ mozjpeg pre-build test failed
  ℹ compiling from source
  ✖ Error: Command failed: /bin/sh -c ./configure --enable-static --disable-shared --disable-dependency-tracking --with-jpeg8  --prefix="/site/node_modules/mozjpeg/vendor" --bindir="/site/node_modules/mozjpeg/vendor" --libdir="/site/node_modules/mozjpeg/vendor"
configure: error: in `/site/node_modules/mozjpeg/18f42ca2-c988-4d9c-9b92-ce6b9a42f50e':
configure: error: C compiler cannot create executables
See `config.log' for more details

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... ./install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking whether make supports the include directive... yes (GNU style)
checking for gcc... gcc
checking whether the C compiler works... no

Full Dockerfile:

# Use gatsby-cli instead of gatsby, roughly 150MB vs 15MB difference!

# Remove dev dependencies that aren't required anymore(not building vips-dev), 
# if a user needs to build for specific package they should build a new docker image extending this one
# python2 is about 40MB and might be useful for packages needing node-gyp to build
# g++ is about 150MB, gcc 85MB
FROM node:10-alpine # 70MB

# Install dev dependencies (gcc 85MB, git 15MB)
RUN set -x \
  && apk add --no-cache \
  gcc \
  git \
  make

# Install gatsby cli and cleanup (adds about 15MB)
ARG GATSBY_VERSION=^2.0.0
RUN set -x \
  && npm install --global gatsby-cli@${GATSBY_VERSION} \
  --no-optional gatsby-cli@${GATSBY_VERSION} \
  && npm cache clean --force

# These let toFormat work for image format conversions with webp
# imagemin plugins support(part of gatsby-plugin-sharp) (adds about 5MB)
# lcms2-dev - jpeg and tiff support
# libpng-dev - png support
RUN set -x \
  && apk add --no-cache \
  lcms2-dev \
  libpng-dev

# mozjpeg support 46MB (also depends on libpng-dev)
RUN set -x \
  && apk add --no-cache \
  autoconf \
  automake \
  libtool \
  nasm

RUN mkdir -p /site
WORKDIR /site
VOLUME /site

EXPOSE 8000

COPY ./entry.sh /
RUN chmod +x /entry.sh
ENTRYPOINT ["/entry.sh"]

EDIT:

While looking into multi-stage build for mozjpeg, it seems that they've recently switched away from autoreconf and use cmake now, so the current npm package is a tad out of date. Thus the dependencies aren't so relevant anymore..