mastodon / mastodon

Your self-hosted, globally interconnected microblogging community
https://joinmastodon.org
GNU Affero General Public License v3.0
47.07k stars 6.97k forks source link

ERROR [stage-1 9/10] during docker-compose build of 4.0.0 #20712

Closed tarasis closed 1 year ago

tarasis commented 1 year ago

Steps to reproduce the problem

I am upgrading from a running Docker install of 3.5.3

I did:

  1. git fetch && git checkout v4.0.0
  2. docker-compose build

Expected behaviour

That the build completes

Actual behaviour

Build failed after 63 minutes with an ERROR [stage-1 9/10]. "failed to solve: executor failed running"

Detailed description

[+] Building 3813.8s (19/20)                                                                                                                                                      
 => [internal] load .dockerignore                                                                                                                                            1.5s
 => => transferring context: 240B                                                                                                                                            0.3s
 => [internal] load build definition from Dockerfile                                                                                                                         1.4s
 => => transferring dockerfile: 4.01kB                                                                                                                                       0.2s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                             10.4s
 => [internal] load build context                                                                                                                                           61.6s
 => => transferring context: 55.86MB                                                                                                                                        60.5s
 => [build-dep 1/7] FROM docker.io/library/ubuntu:20.04@sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61dbd                                             36.6s
 => => resolve docker.io/library/ubuntu:20.04@sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61dbd                                                        0.2s
 => => sha256:b25ef49a40b7797937d0d23eca3b0a41701af6757afca23d504d50826f0b37ce 529B / 529B                                                                                   0.0s
 => => sha256:680e5dfb52c74a1fbc99c2922c8e25b5736e6cd1a3d9430890d52a4f8f44087a 1.46kB / 1.46kB                                                                               0.0s
 => => sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61dbd 1.42kB / 1.42kB                                                                               0.0s
 => => sha256:eaead16dc43bb8811d4ff450935d607f9ba4baffda4fc110cc402fa43f601d83 28.58MB / 28.58MB                                                                             4.1s
 => => extracting sha256:eaead16dc43bb8811d4ff450935d607f9ba4baffda4fc110cc402fa43f601d83                                                                                   12.5s
 => [build-dep 2/7] RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections                                                                      41.1s
 => [build-dep 3/7] RUN ARCH= &&     dpkgArch="$(dpkg --print-architecture)" &&   case "${dpkgArch##*-}" in     amd64) ARCH='x64';;     ppc64el) ARCH='ppc64le';;     s390  99.8s
 => [build-dep 4/7] RUN apt-get update &&   apt-get install -y --no-install-recommends build-essential     bison libyaml-dev libgdbm-dev libreadline-dev libjemalloc-dev   565.4s 
 => [build-dep 5/7] RUN npm install -g npm@latest &&  npm install -g yarn &&  gem install bundler &&  apt-get update &&  apt-get install -y --no-install-recommends git l  116.0s 
 => [build-dep 6/7] COPY Gemfile* package.json yarn.lock /opt/mastodon/                                                                                                      6.8s 
 => [build-dep 7/7] RUN cd /opt/mastodon &&   bundle config set --local deployment 'true' &&   bundle config set --local without 'development test' &&   bundle config s  1393.1s 
 => [stage-1  2/10] COPY --from=build-dep /opt/node /opt/node                                                                                                                6.6s 
 => [stage-1  3/10] COPY --from=build-dep /opt/ruby /opt/ruby                                                                                                               13.9s 
 => [stage-1  4/10] RUN apt-get update &&  echo "Etc/UTC" > /etc/localtime &&  apt-get install -y --no-install-recommends whois wget &&  addgroup --gid 991 mastodon &&  u  18.1s 
 => [stage-1  5/10] RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections                                                                       1.4s 
 => [stage-1  6/10] RUN apt-get update &&   apt-get -y --no-install-recommends install    libssl1.1 libpq5 imagemagick ffmpeg libjemalloc2    libicu66 libidn11 libyaml-0-  96.1s 
 => [stage-1  7/10] COPY --chown=mastodon:mastodon . /opt/mastodon                                                                                                          11.2s 
 => [stage-1  8/10] COPY --from=build-dep --chown=mastodon:mastodon /opt/mastodon /opt/mastodon                                                                            128.4s 
 => ERROR [stage-1  9/10] RUN cd ~ &&  OTP_SECRET=precompile_placeholder SECRET_KEY_BASE=precompile_placeholder rails assets:precompile &&  yarn cache clean              1118.6s 
------                                                                                                                                                                            
 > [stage-1  9/10] RUN cd ~ &&  OTP_SECRET=precompile_placeholder SECRET_KEY_BASE=precompile_placeholder rails assets:precompile &&     yarn cache clean:                         
#0 73.50 yarn install v1.22.19                                                                                                                                                    
#0 74.41 [1/6] Validating package.json...                                                                                                                                         
#0 74.42 [2/6] Resolving packages...                                                                                                                                              
#0 75.77 [3/6] Fetching packages...                                                                                                                                               
#0 123.9 [4/6] Linking dependencies...
#0 124.0 warning Workspaces can only be enabled in private projects.
#0 359.4 [5/6] Building fresh packages...
#0 366.9 [6/6] Cleaning modules...
#0 412.5 Done in 338.83s.
#0 494.8 I, [2022-11-14T22:47:42.547433 #7]  INFO -- : Writing /opt/mastodon/public/assets/doorkeeper/admin/application-a644908e7bab54fb749be0f59fb64a7480bbf9c4c2b79d4a65791cb7ab4d8730.css
#0 497.3 I, [2022-11-14T22:47:45.571652 #7]  INFO -- : Writing /opt/mastodon/public/assets/doorkeeper/admin/application-a644908e7bab54fb749be0f59fb64a7480bbf9c4c2b79d4a65791cb7ab4d8730.css.gz
#0 498.8 I, [2022-11-14T22:47:47.013724 #7]  INFO -- : Writing /opt/mastodon/public/assets/doorkeeper/application-c93dac2ad9d65e3393e0e2c958481e86ef7a5e5b0f6ce406842a7b99b25a4850.css
#0 498.9 I, [2022-11-14T22:47:47.102401 #7]  INFO -- : Writing /opt/mastodon/public/assets/doorkeeper/application-c93dac2ad9d65e3393e0e2c958481e86ef7a5e5b0f6ce406842a7b99b25a4850.css.gz
#0 499.1 I, [2022-11-14T22:47:47.392759 #7]  INFO -- : Writing /opt/mastodon/public/assets/pghero/favicon-db10337a56c45eb43c22ff5019546b520fa22c7281d4d385f235cbca67ed26bb.png
#0 510.1 I, [2022-11-14T22:47:58.345974 #7]  INFO -- : Writing /opt/mastodon/public/assets/pghero/application-a60bf0a452ed064fef3594cf52a4c998712da7c76150f890f4eaa644f59671e4.js
#0 510.1 I, [2022-11-14T22:47:58.348279 #7]  INFO -- : Writing /opt/mastodon/public/assets/pghero/application-a60bf0a452ed064fef3594cf52a4c998712da7c76150f890f4eaa644f59671e4.js.gz
#0 510.7 I, [2022-11-14T22:47:58.904489 #7]  INFO -- : Writing /opt/mastodon/public/assets/pghero/application-c31338f656687c1d733bb0f48d40acd076e24060f3dcff83b34870e4ccc2789d.css
#0 510.7 I, [2022-11-14T22:47:58.904715 #7]  INFO -- : Writing /opt/mastodon/public/assets/pghero/application-c31338f656687c1d733bb0f48d40acd076e24060f3dcff83b34870e4ccc2789d.css.gz
#0 542.8 Compiling...
#0 1102.4 Compilation failed:
#0 1102.4 
------
failed to solve: executor failed running [/bin/bash -o pipefail -c cd ~ 
&&  OTP_SECRET=precompile_placeholder SECRET_KEY_BASE=precompile_placeholder rails assets:precompile 
&&  yarn cache clean]: exit code: 1

Specifications

Mastodon tag 4.0.0 Docker version 20.10.20, build 9fdeb9c3de & then Docker version 20.10.21, build baeda1f82a ARCH LINUX Latest.

afontenot commented 1 year ago

Not an expert, but my understanding is that a lot of asset compilation failures (which is what you're facing here) that don't have a clear error message (nothing after "compilation failed:") are the result of running out of memory. The build process takes quite a lot. Could that be the case here?

tarasis commented 1 year ago

Very possible. It's a 2GB machine with 1GB of Swap. It was getting awfully close on both. Will increase swap file size and try again.

dfloer commented 1 year ago

FWIW, I was trying to run docker-compose build from main at commit 302a58c2 and was getting the same error, but going back to tag v4.0.0rc2 (commit 5187e4e7) didn't have the issue. Similar setup, Archlinux with similar versions of docker (20.10.21).

dfloer commented 1 year ago

I can confirm that I ran out of memory on a VPS with 4GB of RAM and 512MB of swap on that step, and got that error.

ClearlyClaire commented 1 year ago

It indeed sounds like webpack running out of memory.

Annoyingly, when that happens, webpacker has a tendency to pretend it doesn't need to do anything on subsequent calls. One way to address that is to run RAILS_ENV=production bundle exec rails assets:clobber before retrying.

As for being actually able to build the assets, you have a few options:

tarasis commented 1 year ago

Thanks @ClearlyClaire for the options. For doing the compile elsewhere is it as simple as copying the mastodon directory and docker directory and building remotely? Not something I've done before.

I bumped my swap from 1GB to 3GB, and left it to build while I slept. It seems to have completed okay although the compile took ages (67 minutes).

[+] Building 5656.7s (21/21) FINISHED                                                                                                                                             
 => [internal] load build definition from Dockerfile                                                                                                                         2.2s
 => => transferring dockerfile: 32B                                                                                                                                          0.8s
 => [internal] load .dockerignore                                                                                                                                            2.2s
 => => transferring context: 35B                                                                                                                                             0.7s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                              9.0s
 => [build-dep 1/7] FROM docker.io/library/ubuntu:20.04@sha256:450e066588f42ebe1551f3b1a535034b6aa46cd936fe7f2c6b0d72997ec61dbd                                              0.0s
 => [internal] load build context                                                                                                                                            3.4s
 => => transferring context: 454.95kB                                                                                                                                        3.3s
 => CACHED [build-dep 2/7] RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections                                                                0.0s
 => CACHED [build-dep 3/7] RUN ARCH= &&     dpkgArch="$(dpkg --print-architecture)" &&   case "${dpkgArch##*-}" in     amd64) ARCH='x64';;     ppc64el) ARCH='ppc64le';;     0.0s
 => CACHED [build-dep 4/7] RUN apt-get update &&   apt-get install -y --no-install-recommends build-essential     bison libyaml-dev libgdbm-dev libreadline-dev libjemalloc  0.0s
 => CACHED [build-dep 5/7] RUN npm install -g npm@latest &&  npm install -g yarn &&  gem install bundler &&  apt-get update &&  apt-get install -y --no-install-recommends   0.0s
 => CACHED [build-dep 6/7] COPY Gemfile* package.json yarn.lock /opt/mastodon/                                                                                               0.0s
 => CACHED [build-dep 7/7] RUN cd /opt/mastodon &&   bundle config set --local deployment 'true' &&   bundle config set --local without 'development test' &&   bundle conf  0.0s
 => CACHED [stage-1  2/10] COPY --from=build-dep /opt/node /opt/node                                                                                                         0.0s
 => CACHED [stage-1  3/10] COPY --from=build-dep /opt/ruby /opt/ruby                                                                                                         0.0s
 => CACHED [stage-1  4/10] RUN apt-get update &&  echo "Etc/UTC" > /etc/localtime &&  apt-get install -y --no-install-recommends whois wget &&  addgroup --gid 991 mastodon  0.0s
 => CACHED [stage-1  5/10] RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections                                                                0.0s
 => CACHED [stage-1  6/10] RUN apt-get update &&   apt-get -y --no-install-recommends install    libssl1.1 libpq5 imagemagick ffmpeg libjemalloc2    libicu66 libidn11 liby  0.0s
 => [stage-1  7/10] COPY --chown=mastodon:mastodon . /opt/mastodon                                                                                                          32.4s
 => [stage-1  8/10] COPY --from=build-dep --chown=mastodon:mastodon /opt/mastodon /opt/mastodon                                                                            643.7s
 => [stage-1  9/10] RUN cd ~ &&  OTP_SECRET=precompile_placeholder SECRET_KEY_BASE=precompile_placeholder rails assets:precompile &&  yarn cache clean                    4063.5s
 => [stage-1 10/10] WORKDIR /opt/mastodon                                                                                                                                    1.4s 
 => exporting to image                                                                                                                                                     897.5s 
 => => exporting layers                                                                                                                                                    896.2s 
 => => writing image sha256:ae4a59dfbf1317da8e7b767204789b01c0d4cda8f5326187f349081f4b0936b9                                                                                 0.1s 
 => => naming to docker.io/tootsuite/mastodon