voc / voctoweb

voctoweb – the frontend and backend software behind media.ccc.de
GNU General Public License v3.0
189 stars 58 forks source link

Docker setup fails with missing dependencies #589

Open flauschzelle opened 2 years ago

flauschzelle commented 2 years ago

I tried to install the docker setup as described here in the README. The first command of the setup script (docker-compose build) fails with the following message:

+ apk add --virtual .voctoweb-rundeps so:libc.musl-x86_64.so.1 so:libc.so.6 so:libcrypto.so.1.1 so:libdl.so.2 so:libffi.so.8 so:libgcc_s.so.1 so:libm.so.6 so:libpq.so.5 so:libpthread.so.0 so:libssl.so.1.1 so:libstdc++.so.6
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
ERROR: unable to select packages:
  so:libc.so.6 (no such package):
    required by: .voctoweb-rundeps-20220108.131144[so:libc.so.6]
  so:libdl.so.2 (no such package):
    required by: .voctoweb-rundeps-20220108.131144[so:libdl.so.2]
  so:libm.so.6 (no such package):
    required by: .voctoweb-rundeps-20220108.131144[so:libm.so.6]
  so:libpthread.so.0 (no such package):
    required by: .voctoweb-rundeps-20220108.131144[so:libpthread.so.0]
ERROR: Service 'sidekiq' failed to build: The command '/bin/sh -c set -eux;     apk add --no-cache --virtual .build-deps                g++        git              gcc             libxml2-dev             libxslt-dev             make            musl-dev                patch           postgresql-dev      ;               gem install bundler --version 2.2.11;   bundle install;         rm -r ~/.bundle;                runDeps="$(             scanelf --needed --nobanner --format '%n#p' --recursive /usr/local/bundle/gems                      | tr ',' '\n'                   | sort -u          | awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }'  )";     apk add --virtual .voctoweb-rundeps $runDeps;   apk del .build-deps' returned a non-zero code: 5

I assume that an update of the alpine container version might have broken some of the dependencies?

Can you reproduce that or do you need more information from me?

Here is the complete output:

click to open ``` $ docker-compose build ssh-server uses an image, skipping postgres uses an image, skipping redis uses an image, skipping Building sidekiq Step 1/7 : FROM ruby:2.6-alpine 2.6-alpine: Pulling from library/ruby 59bf1c3509f3: Already exists ecfdfc99c764: Pull complete dfcec1f1cb7c: Pull complete 1e13173002d8: Pull complete 6e0f30c9e5cc: Pull complete Digest: sha256:a5f53bb336c910f5b30d8df7cb9c01e26f61dc43d8757afdeff0c0c834ee929f Status: Downloaded newer image for ruby:2.6-alpine ---> d11d7eff8033 Step 2/7 : RUN set -eux; apk add --no-cache dumb-init nodejs tzdata ; ---> Running in 05ca271c2a87 + apk add --no-cache dumb-init nodejs tzdata fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz (1/6) Installing dumb-init (1.2.5-r1) (2/6) Installing nghttp2-libs (1.46.0-r0) (3/6) Installing brotli-libs (1.0.9-r5) (4/6) Installing c-ares (1.18.1-r0) (5/6) Installing nodejs (16.13.1-r0) (6/6) Installing tzdata (2021e-r0) Executing busybox-1.34.1-r3.trigger OK: 63 MiB in 42 packages Removing intermediate container 05ca271c2a87 ---> 4c5b29a2afbd Step 3/7 : WORKDIR /voctoweb ---> Running in 80aa11ddcd8b Removing intermediate container 80aa11ddcd8b ---> 89379bdd657f Step 4/7 : ENV BUNDLE_FORCE_RUBY_PLATFORM 1 ---> Running in 462dbb9b4ea5 Removing intermediate container 462dbb9b4ea5 ---> 1847b7380708 Step 5/7 : ENV BUNDLE_WITHOUT "development:test" ---> Running in 64fffa8319d6 Removing intermediate container 64fffa8319d6 ---> f3315e2bace1 Step 6/7 : COPY Gemfile Gemfile.lock /voctoweb/ ---> bb58309a7226 Step 7/7 : RUN set -eux; apk add --no-cache --virtual .build-deps g++ git gcc libxml2-dev libxslt-dev make musl-dev patch postgresql-dev ; gem install bundler --version 2.2.11; bundle install; rm -r ~/.bundle; runDeps="$( scanelf --needed --nobanner --format '%n#p' --recursive /usr/local/bundle/gems | tr ',' '\n' | sort -u | awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }' )"; apk add --virtual .voctoweb-rundeps $runDeps; apk del .build-deps ---> Running in 00ca9696e618 + apk add --no-cache --virtual .build-deps g++ git gcc libxml2-dev libxslt-dev make musl-dev patch postgresql-dev fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz (1/41) Installing binutils (2.37-r3) (2/41) Installing libgomp (10.3.1_git20211027-r0) (3/41) Installing libatomic (10.3.1_git20211027-r0) (4/41) Installing libgphobos (10.3.1_git20211027-r0) (5/41) Installing isl22 (0.22-r0) (6/41) Installing mpfr4 (4.1.0-r0) (7/41) Installing mpc1 (1.2.1-r0) (8/41) Installing gcc (10.3.1_git20211027-r0) (9/41) Installing musl-dev (1.2.2-r7) (10/41) Installing libc-dev (0.7.2-r3) (11/41) Installing g++ (10.3.1_git20211027-r0) (12/41) Upgrading libcrypto1.1 (1.1.1l-r7 -> 1.1.1l-r8) (13/41) Upgrading libssl1.1 (1.1.1l-r7 -> 1.1.1l-r8) (14/41) Installing libcurl (7.80.0-r0) (15/41) Installing expat (2.4.1-r0) (16/41) Installing pcre2 (10.39-r0) (17/41) Installing git (2.34.1-r0) (18/41) Installing xz-libs (5.2.5-r0) (19/41) Installing xz-dev (5.2.5-r0) (20/41) Installing libxml2 (2.9.12-r2) (21/41) Installing libxml2-dev (2.9.12-r2) (22/41) Installing libgpg-error (1.42-r1) (23/41) Installing libgcrypt (1.9.4-r0) (24/41) Installing libxslt (1.1.34-r1) (25/41) Installing libxslt-dev (1.1.34-r1) (26/41) Installing make (4.3-r0) (27/41) Installing patch (2.7.6-r7) (28/41) Installing libpq (14.1-r5) (29/41) Installing openssl-dev (1.1.1l-r8) (30/41) Installing libpq-dev (14.1-r5) (31/41) Installing libecpg (14.1-r5) (32/41) Installing libecpg-dev (14.1-r5) (33/41) Installing llvm12-libs (12.0.1-r0) (34/41) Installing clang-libs (12.0.1-r1) (35/41) Installing clang (12.0.1-r1) (36/41) Installing icu-libs (69.1-r1) (37/41) Installing icu (69.1-r1) (38/41) Installing icu-dev (69.1-r1) (39/41) Installing llvm12 (12.0.1-r0) (40/41) Installing postgresql14-dev (14.1-r5) (41/41) Installing .build-deps (20220108.130442) Executing busybox-1.34.1-r3.trigger OK: 581 MiB in 81 packages + gem install bundler --version 2.2.11 Successfully installed bundler-2.2.11 1 gem installed + bundle install Fetching gem metadata from https://rubygems.org/......... Fetching rake 13.0.6 Installing rake 13.0.6 Fetching thread_safe 0.3.6 Fetching minitest 5.14.4 Fetching builder 3.2.4 Fetching concurrent-ruby 1.1.9 Installing builder 3.2.4 Fetching erubi 1.10.0 Installing minitest 5.14.4 Installing thread_safe 0.3.6 Fetching racc 1.5.2 Fetching crass 1.0.6 Installing concurrent-ruby 1.1.9 Installing erubi 1.10.0 Fetching rack 2.2.3 Fetching nio4r 2.5.7 Installing crass 1.0.6 Fetching websocket-extensions 0.1.5 Installing rack 2.2.3 Installing racc 1.5.2 with native extensions Installing nio4r 2.5.7 with native extensions Fetching mini_mime 1.1.0 Installing websocket-extensions 0.1.5 Installing mini_mime 1.1.0 Fetching ruby2_keywords 0.0.4 Installing ruby2_keywords 0.0.4 Fetching method_source 1.0.0 Fetching formtastic_i18n 0.7.0 Fetching thor 1.1.0 Fetching kaminari-core 1.2.1 Installing method_source 1.0.0 Installing formtastic_i18n 0.7.0 Fetching arel 9.0.0 Installing kaminari-core 1.2.1 Installing thor 1.1.0 Fetching marcel 1.0.1 Fetching net-ssh 6.1.0 Fetching api-pagination 4.8.2 Installing arel 9.0.0 Installing marcel 1.0.1 Installing api-pagination 4.8.2 Fetching google-protobuf 3.17.3 (x86_64-linux) Fetching google-protobuf 3.17.3 (universal-darwin) Fetching graphql 1.12.14 Installing net-ssh 6.1.0 Fetching execjs 2.8.1 Installing google-protobuf 3.17.3 (x86_64-linux) Installing execjs 2.8.1 Installing graphql 1.12.14 Fetching bcrypt 3.1.16 Installing google-protobuf 3.17.3 (universal-darwin) Fetching ffi 1.15.3 Using bundler 2.2.11 Fetching coffee-script-source 1.12.2 Installing bcrypt 3.1.16 with native extensions Installing coffee-script-source 1.12.2 Installing ffi 1.15.3 with native extensions Fetching connection_pool 2.2.5 Fetching orm_adapter 0.5.0 Installing connection_pool 2.2.5 Installing orm_adapter 0.5.0 Fetching faraday-em_http 1.0.0 Fetching dotenv 2.7.6 Fetching multi_json 1.15.0 Fetching faraday-em_synchrony 1.0.0 Installing faraday-em_http 1.0.0 Installing dotenv 2.7.6 Installing multi_json 1.15.0 Installing faraday-em_synchrony 1.0.0 Fetching faraday-excon 1.1.0 Fetching faraday-httpclient 1.0.1 Fetching faraday-net_http 1.0.1 Fetching faraday-net_http_persistent 1.2.0 Installing faraday-excon 1.1.0 Installing faraday-net_http_persistent 1.2.0 Installing faraday-net_http 1.0.1 Fetching faraday-patron 1.0.0 Installing faraday-httpclient 1.0.1 Fetching multipart-post 2.1.1 Fetching hashie 4.1.0 Fetching elasticsearch-rails 6.1.1 Installing faraday-patron 1.0.0 Fetching foreman 0.87.2 Installing multipart-post 2.1.1 Fetching temple 0.8.2 Installing hashie 4.1.0 Fetching tilt 2.0.10 Installing temple 0.8.2 Installing foreman 0.87.2 Fetching rb-fsevent 0.11.0 Installing tilt 2.0.10 Fetching pg 1.2.3 Fetching rdoc 6.3.2 Installing elasticsearch-rails 6.1.1 Installing rb-fsevent 0.11.0 Fetching redcarpet 3.5.1 Installing pg 1.2.3 with native extensions Fetching redis 4.3.1 Installing rdoc 6.3.2 Installing redcarpet 3.5.1 with native extensions Installing redis 4.3.1 Fetching rexml 3.2.5 Installing rexml 3.2.5 Fetching search_object 1.2.4 Fetching yajl-ruby 1.4.1 Fetching turbolinks-source 5.2.0 Fetching tzinfo 1.2.9 Installing turbolinks-source 5.2.0 Fetching aasm 5.2.0 Installing search_object 1.2.4 Installing yajl-ruby 1.4.1 with native extensions Installing tzinfo 1.2.9 Installing aasm 5.2.0 Fetching i18n 1.8.10 Fetching rack-test 1.1.0 Installing i18n 1.8.10 Installing rack-test 1.1.0 Fetching warden 1.2.9 Fetching request_store 1.5.0 Fetching sprockets 4.0.2 Fetching rack-cors 1.1.1 Installing request_store 1.5.0 Installing warden 1.2.9 Fetching websocket-driver 0.7.5 Fetching mail 2.7.1 Installing sprockets 4.0.2 Installing rack-cors 1.1.1 Fetching puma 5.2.2 Fetching nokogiri 1.11.7 (x86_64-linux) Installing websocket-driver 0.7.5 with native extensions Installing puma 5.2.2 with native extensions Installing mail 2.7.1 Installing nokogiri 1.11.7 (x86_64-linux) Fetching nokogiri 1.11.7 (x86_64-darwin) Fetching autoprefixer-rails 10.2.5.1 Fetching net-scp 3.0.0 Fetching uglifier 4.2.0 Installing net-scp 3.0.0 Fetching coffee-script 2.4.1 Installing uglifier 4.2.0 Installing autoprefixer-rails 10.2.5.1 Fetching apollo-federation 1.1.5 Fetching graphql-query-resolver 0.2.0 Installing coffee-script 2.4.1 Fetching sassc 2.4.0 Installing apollo-federation 1.1.5 Fetching get_process_mem 0.2.7 Installing graphql-query-resolver 0.2.0 Fetching rb-inotify 0.10.1 Installing get_process_mem 0.2.7 Installing sassc 2.4.0 with native extensions Fetching elasticsearch-api 6.8.3 Installing rb-inotify 0.10.1 Fetching faraday 1.5.1 Installing elasticsearch-api 6.8.3 Installing faraday 1.5.1 Installing nokogiri 1.11.7 (x86_64-darwin) Fetching haml 5.2.1 Fetching sdoc 2.2.0 Fetching redis-store 1.9.0 Fetching sidekiq 6.2.1 Installing redis-store 1.9.0 Installing haml 5.2.1 Fetching ruby-graphviz 1.2.5 Installing sdoc 2.2.0 Fetching turbolinks 5.2.1 Installing sidekiq 6.2.1 Fetching search_object_graphql 1.0.1 Installing ruby-graphviz 1.2.5 Installing turbolinks 5.2.1 Fetching activesupport 5.2.6 Fetching loofah 2.10.0 Installing search_object_graphql 1.0.1 Fetching sshkit 1.21.2 Fetching puma_worker_killer 0.3.1 Installing loofah 2.10.0 Installing activesupport 5.2.6 Installing puma_worker_killer 0.3.1 Fetching listen 3.6.0 Installing sshkit 1.21.2 Fetching elasticsearch-transport 6.8.3 Installing listen 3.6.0 Fetching bootstrap-sass 3.4.1 Fetching redis-rack 2.1.3 Installing elasticsearch-transport 6.8.3 Fetching rails-html-sanitizer 1.3.0 Installing redis-rack 2.1.3 Fetching airbrussh 1.4.0 Installing bootstrap-sass 3.4.1 Fetching elasticsearch 6.8.3 Installing rails-html-sanitizer 1.3.0 Installing airbrussh 1.4.0 Fetching rails-dom-testing 2.0.3 Installing elasticsearch 6.8.3 Fetching globalid 0.4.2 Fetching arbre 1.4.0 Installing rails-dom-testing 2.0.3 Fetching activemodel 5.2.6 Fetching jbuilder 2.11.2 Installing globalid 0.4.2 Installing arbre 1.4.0 Fetching redis-activesupport 5.2.1 Installing jbuilder 2.11.2 Fetching utf8-cleaner 1.0.0 Installing activemodel 5.2.6 Fetching elasticsearch-model 6.1.1 Fetching capistrano 3.16.0 Installing redis-activesupport 5.2.1 Installing utf8-cleaner 1.0.0 Fetching actionview 5.2.6 Fetching activejob 5.2.6 Installing elasticsearch-model 6.1.1 Installing capistrano 3.16.0 Installing activejob 5.2.6 Installing actionview 5.2.6 Fetching activerecord 5.2.6 Fetching kaminari-actionview 1.2.1 Fetching actionpack 5.2.6 Installing kaminari-actionview 1.2.1 Installing activerecord 5.2.6 Installing actionpack 5.2.6 Fetching formtastic 4.0.0 Fetching actioncable 5.2.6 Fetching actionmailer 5.2.6 Fetching has_scope 0.8.0 Installing actionmailer 5.2.6 Installing actioncable 5.2.6 Installing has_scope 0.8.0 Installing formtastic 4.0.0 Fetching railties 5.2.6 Fetching sprockets-rails 3.2.2 Fetching redis-actionpack 5.2.0 Fetching kaminari-activerecord 1.2.1 Installing sprockets-rails 3.2.2 Installing redis-actionpack 5.2.0 Fetching ransack 2.4.2 Installing kaminari-activerecord 1.2.1 Fetching activestorage 5.2.6 Installing railties 5.2.6 Fetching goldiloader 4.1.0 Installing goldiloader 4.1.0 Installing activestorage 5.2.6 Installing ransack 2.4.2 Fetching exception_notification 4.4.3 Installing exception_notification 4.4.3 Fetching redis-rails 5.0.2 Installing redis-rails 5.0.2 Fetching kaminari 1.2.1 Fetching jquery-rails 4.4.0 Fetching coffee-rails 5.0.0 Fetching responders 3.0.1 Installing kaminari 1.2.1 Installing coffee-rails 5.0.0 Fetching dotenv-rails 2.7.6 Installing responders 3.0.1 Installing jquery-rails 4.4.0 Fetching graphiql-rails 1.7.0 Installing dotenv-rails 2.7.6 Fetching lograge 0.11.2 Fetching rails 5.2.6 Fetching sassc-rails 2.1.2 Installing lograge 0.11.2 Installing rails 5.2.6 Fetching tinymce-rails 5.8.2 Installing sassc-rails 2.1.2 Fetching inherited_resources 1.13.0 Fetching devise 4.8.0 Installing inherited_resources 1.13.0 Installing devise 4.8.0 Fetching sass-rails 6.0.0 Installing tinymce-rails 5.8.2 Installing sass-rails 6.0.0 Fetching activeadmin 2.9.0 Installing activeadmin 2.9.0 Installing graphiql-rails 1.7.0 Fetching https://github.com/njh/ruby-mqtt.git Bundle complete! 53 Gemfile dependencies, 138 gems now installed. Gems in the groups development and test were not installed. Use `bundle info [gemname]` to see where a bundled gem is installed. Post-install message from ruby-graphviz: You need to install GraphViz (https://graphviz.org) to use this Gem. For more information about Ruby-Graphviz : * Doc: https://rdoc.info/github/glejeune/Ruby-Graphviz * Sources: https://github.com/glejeune/Ruby-Graphviz * Issues: https://github.com/glejeune/Ruby-Graphviz/issues + rm -r /root/.bundle + scanelf --needed --nobanner --format '%n#p' --recursive /usr/local/bundle/gems + sort -u + tr , '\n' + awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }' + runDeps='so:libc.musl-x86_64.so.1 so:libc.so.6 so:libcrypto.so.1.1 so:libdl.so.2 so:libffi.so.8 so:libgcc_s.so.1 so:libm.so.6 so:libpq.so.5 so:libpthread.so.0 so:libssl.so.1.1 so:libstdc++.so.6' + apk add --virtual .voctoweb-rundeps so:libc.musl-x86_64.so.1 so:libc.so.6 so:libcrypto.so.1.1 so:libdl.so.2 so:libffi.so.8 so:libgcc_s.so.1 so:libm.so.6 so:libpq.so.5 so:libpthread.so.0 so:libssl.so.1.1 so:libstdc++.so.6 fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz ERROR: unable to select packages: so:libc.so.6 (no such package): required by: .voctoweb-rundeps-20220108.131144[so:libc.so.6] so:libdl.so.2 (no such package): required by: .voctoweb-rundeps-20220108.131144[so:libdl.so.2] so:libm.so.6 (no such package): required by: .voctoweb-rundeps-20220108.131144[so:libm.so.6] so:libpthread.so.0 (no such package): required by: .voctoweb-rundeps-20220108.131144[so:libpthread.so.0] ERROR: Service 'sidekiq' failed to build: The command '/bin/sh -c set -eux; apk add --no-cache --virtual .build-deps g++ git gcc libxml2-dev libxslt-dev make musl-dev patch postgresql-dev ; gem install bundler --version 2.2.11; bundle install; rm -r ~/.bundle; runDeps="$( scanelf --needed --nobanner --format '%n#p' --recursive /usr/local/bundle/gems | tr ',' '\n' | sort -u | awk 'system("[ -e /usr/local/lib/" $1 " ]") == 0 { next } { print "so:" $1 }' )"; apk add --virtual .voctoweb-rundeps $runDeps; apk del .build-deps' returned a non-zero code: 5 ```
blinry commented 2 years ago

Deleting the second-to-last line from the Dockerfile (apk add --virtual .voctoweb-rundeps $runDeps), seems to avoid that problem, and the build step succeeds for me! I think that line tries to collect runtime dependencies in a package group, and the musl-instead-of-libc situation causes problems?

After the build step, I run into two more problems:

$ docker-compose run voctoweb rake db:setup
rake aborted!
LoadError: cannot load such file -- nokogiri/nokogiri
...
Caused by:
LoadError: cannot load such file -- /usr/local/bundle/gems/nokogiri-1.12.4-x86_64-darwin/lib/nokogiri/2.6/nokogiri
$ docker-compose run voctoweb bin/update-data
== Downloading Data-Dump ==
sh: curl: not found
tar: invalid magic
tar: short read

== Command ["curl https://media.ccc.de/system/voctoweb.dump.tar.gz | tar xvz"] failed ==
flauschzelle commented 2 years ago

When i delete the line you mentioned, the docker-compose build step completes successfully, thanks :)

The next step (docker-compose up -d postgres) also completes without problems:

$ docker-compose up -d postgres
Creating network "voctoweb_default" with the default driver
Creating voctoweb_postgres_1 ... 
Creating voctoweb_postgres_1 ... done

Then in the docker-compose run voctoweb rake db:setup step it fails again, but I get different error messages O.o

rake aborted!
LoadError: Error loading the 'postgresql' Active Record adapter. Missing a gem it depends on? Error loading shared library libpq.so.5: No such file or directory (needed by /usr/local/bundle/gems/pg-1.2.3/lib/pg_ext.so) - /usr/local/bundle/gems/pg-1.2.3/lib/pg_ext.so

...

Caused by:
LoadError: Error loading shared library libpq.so.5: No such file or directory (needed by /usr/local/bundle/gems/pg-1.2.3/lib/pg_ext.so) - /usr/local/bundle/gems/pg-1.2.3/lib/pg_ext.so

Here is the full output:

click to open ``` $ docker-compose run voctoweb rake db:setup Pulling redis (redis:5-alpine)... 5-alpine: Pulling from library/redis 59bf1c3509f3: Already exists 719adce26c52: Pull complete b8f35e378c31: Pull complete 1c2c4f440f7a: Pull complete 51c042fa539b: Pull complete 8cfef0002c8a: Pull complete Digest: sha256:a04b57e05b784a406bc818f6b450233328c4115a1e1f104d84ce916eb1583925 Status: Downloaded newer image for redis:5-alpine Pulling elasticsearch (elasticsearch:6.8.6)... 6.8.6: Pulling from library/elasticsearch ab5ef0e58194: Pull complete 30cb0e53690c: Pull complete 1fbe49af2e05: Pull complete d5c2f84a2853: Pull complete 5abe907f1981: Pull complete af265c640f77: Pull complete 02437bb6d816: Pull complete Digest: sha256:be763d0b98b58a963f0496677be546211ec9aa336b0981ede98cec63325f55ef Status: Downloaded newer image for elasticsearch:6.8.6 Creating voctoweb_redis_1 ... Starting voctoweb_postgres_1 ... Starting voctoweb_postgres_1 ... done Creating voctoweb_elasticsearch_1 Creating voctoweb_redis_1 ... done Creating voctoweb_sidekiq_1 ... Creating voctoweb_sidekiq_1 ... done rake aborted! LoadError: Error loading the 'postgresql' Active Record adapter. Missing a gem it depends on? Error loading shared library libpq.so.5: No such file or directory (needed by /usr/local/bundle/gems/pg-1.2.3/lib/pg_ext.so) - /usr/local/bundle/gems/pg-1.2.3/lib/pg_ext.so /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/pg-1.2.3/lib/pg.rb:5:in `' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:5:in `' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/connection_adapters/connection_specification.rb:191:in `spec' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:975:in `establish_connection' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/connection_handling.rb:60:in `establish_connection' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/railtie.rb:136:in `block (2 levels) in ' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:71:in `instance_eval' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:71:in `block in execute_hook' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:62:in `with_execution_control' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:67:in `execute_hook' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:43:in `block in on_load' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:42:in `each' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:42:in `on_load' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/railtie.rb:132:in `block in ' /usr/local/bundle/gems/railties-5.2.6/lib/rails/initializable.rb:32:in `instance_exec' /usr/local/bundle/gems/railties-5.2.6/lib/rails/initializable.rb:32:in `run' /usr/local/bundle/gems/railties-5.2.6/lib/rails/initializable.rb:61:in `block in run_initializers' /usr/local/bundle/gems/railties-5.2.6/lib/rails/initializable.rb:60:in `run_initializers' /usr/local/bundle/gems/railties-5.2.6/lib/rails/application.rb:361:in `initialize!' /voctoweb/config/environment.rb:5:in `' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/railties-5.2.6/lib/rails/application.rb:337:in `require_environment!' /usr/local/bundle/gems/railties-5.2.6/lib/rails/application.rb:520:in `block in run_tasks_blocks' /usr/local/bundle/gems/rake-13.0.6/exe/rake:27:in `' Caused by: LoadError: Error loading shared library libpq.so.5: No such file or directory (needed by /usr/local/bundle/gems/pg-1.2.3/lib/pg_ext.so) - /usr/local/bundle/gems/pg-1.2.3/lib/pg_ext.so /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/pg-1.2.3/lib/pg.rb:5:in `' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb:5:in `' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/connection_adapters/connection_specification.rb:191:in `spec' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:975:in `establish_connection' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/connection_handling.rb:60:in `establish_connection' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/railtie.rb:136:in `block (2 levels) in ' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:71:in `instance_eval' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:71:in `block in execute_hook' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:62:in `with_execution_control' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:67:in `execute_hook' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:43:in `block in on_load' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:42:in `each' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/lazy_load_hooks.rb:42:in `on_load' /usr/local/bundle/gems/activerecord-5.2.6/lib/active_record/railtie.rb:132:in `block in ' /usr/local/bundle/gems/railties-5.2.6/lib/rails/initializable.rb:32:in `instance_exec' /usr/local/bundle/gems/railties-5.2.6/lib/rails/initializable.rb:32:in `run' /usr/local/bundle/gems/railties-5.2.6/lib/rails/initializable.rb:61:in `block in run_initializers' /usr/local/bundle/gems/railties-5.2.6/lib/rails/initializable.rb:60:in `run_initializers' /usr/local/bundle/gems/railties-5.2.6/lib/rails/application.rb:361:in `initialize!' /voctoweb/config/environment.rb:5:in `' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `block in require' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:257:in `load_dependency' /usr/local/bundle/gems/activesupport-5.2.6/lib/active_support/dependencies.rb:291:in `require' /usr/local/bundle/gems/railties-5.2.6/lib/rails/application.rb:337:in `require_environment!' /usr/local/bundle/gems/railties-5.2.6/lib/rails/application.rb:520:in `block in run_tasks_blocks' /usr/local/bundle/gems/rake-13.0.6/exe/rake:27:in `' Tasks: TOP => db:setup => db:schema:load_if_ruby => db:create => db:load_config => environment (See full trace by running task with --trace) ```
blinry commented 2 years ago

Strange that we get different error messages! I cleared all of my images, containers, and volumes, and had Docker reinstall them, but with the same result.

But I got another step further!

Then I ran docker-compose build again. This allows me to run the rake db:setup step successfully. Now, the bin/update-data step fails, without any helpful hint as to why:

$ docker-compose run voctoweb bin/update-data
[+] Running 4/0
 ⠿ Container voctoweb-redis-1          Running                                                                                                  0.0s
 ⠿ Container voctoweb-postgres-1       Running                                                                                                  0.0s
 ⠿ Container voctoweb-sidekiq-1        Running                                                                                                  0.0s
 ⠿ Container voctoweb-elasticsearch-1  Running                                                                                                  0.0s
== Downloading Data-Dump ==
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0tmp/fixtures/
tmp/fixtures/event_view_counts.yml
tmp/fixtures/news.yml
tmp/fixtures/web_feeds.yml
 81 88.6M   81 72.2M    0     0  11.9M      0  0:00:07  0:00:06  0:00:01 12.2Mtmp/fixtures/conferences.yml
tmp/fixtures/recordings.yml
tmp/fixtures/events.yml
100 88.6M  100 88.6M    0     0  12.0M      0  0:00:07  0:00:07 --:--:-- 12.3M

== Updating database ==

== Command ["FIXTURES_DIR=../../tmp/fixtures bin/rake db:fixtures:load_jsonb"] failed ==

docker-compose up voctoweb works at this point, but it doesn't have any data, so it's not super helpful.

@J0WI, you were the last one touching the Dockerfile – could you check whether the Docker setup still works for you? If not, maybe you can use the information in this issue thread to help figure out how to fix it?

blinry commented 2 years ago

I noticed that when bin/update-data runs the rake db:fixtures:load_jsonb step, it eats all my 12 GB of free RAM, and that's probably why it fails. Huh.

J0WI commented 2 years ago

Unfortunately bundler, respectively nokogiri, ships prebuild binaries that break on various platforms that do not match their ABI: https://github.com/rubygems/rubygems/issues/3174

The proper fix is to revert this line in https://github.com/voc/voctoweb/commit/02afafd46e06c33cbc9855926917764f41fcd813#diff-89cade48462044ee1b672dc5f4c3ec250fbd29effcd8932096a23c1283c6731fL376-R382 to build the gem against whatever platform you are running this. I've opened a PR at #590.