search

Shopify / bootsnap

Boot large Ruby/Rails apps faster
MIT License
2.67k stars 179 forks source link

/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found #429

Closed slondr closed 1 year ago

slondr commented 1 year ago

I made sure the issue is in bootsnap

Yep (also, btw, there is a typo in the GitHub issue template which appears right after this header)

Steps to reproduce

Run any Rails command after installing bootsnap via bundler

Expected behavior

In this case: I ran rails server and expected my server to start

Actual behavior

Application crashed almost immediately, with a big error message starting with LoadError originating in bootsnap.so. I posted the full error message at the bottom of this issue.

System configuration

Bootsnap version: 1.15.0

Ruby version: 2.7.5

Rails version: 6.0.5

Full error message

Puma starting in single mode...
* Puma version: 5.6.5 (ruby 2.7.5-p203) ("Birdie's Version")
*  Min threads: 0
*  Max threads: 5
*  Environment: development
*          PID: 192898
! Unable to load application: LoadError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so) - /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so
Traceback (most recent call last):
    31: from /home/user/.rvm/gems/ruby-2.7.5/bin/puma:23:in `<main>'
    30: from /home/user/.rvm/gems/ruby-2.7.5/bin/puma:23:in `load'
    29: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/bin/puma:10:in `<top (required)>'
    28: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/cli.rb:81:in `run'
    27: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/launcher.rb:193:in `run'
    26: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/single.rb:44:in `run'
    25: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/runner.rb:150:in `load_and_bind'
    24: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/configuration.rb:270:in `app'
    23: from /home/user/.rvm/gems/ruby-2.7.5/gems/puma-5.6.5/lib/puma/configuration.rb:348:in `load_rackup'
    22: from /home/user/.rvm/gems/ruby-2.7.5/gems/rack-2.2.3.1/lib/rack/builder.rb:66:in `parse_file'
    21: from /home/user/.rvm/gems/ruby-2.7.5/gems/rack-2.2.3.1/lib/rack/builder.rb:105:in `load_file'
    20: from /home/user/.rvm/gems/ruby-2.7.5/gems/rack-2.2.3.1/lib/rack/builder.rb:116:in `new_from_string'
    19: from /home/user/.rvm/gems/ruby-2.7.5/gems/rack-2.2.3.1/lib/rack/builder.rb:116:in `eval'
    18: from config.ru:3:in `block in <main>'
    17: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    16: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    15: from /home/user/dev/core/config/environment.rb:2:in `<top (required)>'
    14: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    13: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    12: from /home/user/dev/core/config/application.rb:1:in `<top (required)>'
    11: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
    10: from /home/user/.rvm/rubies/ruby-2.7.5/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
     9: from /home/user/dev/core/config/boot.rb:4:in `<top (required)>'
     8: from /home/user/dev/core/config/boot.rb:4:in `require'
     7: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/setup.rb:5:in `<top (required)>'
     6: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap.rb:100:in `default_setup'
     5: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap.rb:57:in `setup'
     4: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache.rb:16:in `setup'
     3: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache.rb:16:in `require_relative'
     2: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache/iseq.rb:3:in `<top (required)>'
     1: from /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require': /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so) - /home/user/.rvm/gems/ruby-2.7.5/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so (LoadError)
casperisfine commented 1 year ago

Interesting. Likely caused by https://github.com/Shopify/bootsnap/commit/0de36eeb27fa868a323a57a4c63de640809267db, but I wonder if that's even needed.

gabrielrinaldi commented 1 year ago

I am also having the same issue, on my Mac it runs fine, but when dockerizing it it fails with the same message. For some reason building the docker image on my Mac does not cause the issue, but it does on the CI machine.

gabrielrinaldi commented 1 year ago

After more digging I think this is related to the new release of the images, which only has GLIBC_2.35, would that make sense?

casperisfine commented 1 year ago

Yeah that would male sense. if the base image is public would be nice to share it, that would help confirm the bug on my side.

That said I really don't see why we're put the whole glibc in the cache key. I'm pretty sure we can just remove that code, but I need to dig in the history to confirm.

casperisfine commented 1 year ago

Hum, wait, something super weird here. We remove the GLIBC version check in 1.12.0 https://github.com/Shopify/bootsnap/blob/main/CHANGELOG.md#1120

casperisfine commented 1 year ago

Do you re-use your bundler cache across docker image builds? I'm thinking you might need to recompile bootsnap.

gabrielrinaldi commented 1 year ago

This is my docker file:

# Base image
ARG RUBY_VERSION=3.1.3
ARG VARIANT=slim-bullseye

FROM ruby:${RUBY_VERSION}-${VARIANT} as base

ARG BUNDLE_PATH=vendor/bundle
ARG BUNDLE_WITHOUT=development:test
ARG RAILS_ENV=production

ENV BUNDLE_PATH ${BUNDLE_PATH}
ENV BUNDLE_WITHOUT ${BUNDLE_WITHOUT}
ENV RAILS_ENV=${RAILS_ENV}

RUN mkdir /app

WORKDIR /app

RUN mkdir -p tmp/pids

# Build dependencies
FROM base as build_deps

ARG DEV_PACKAGES="git build-essential libpq-dev wget vim curl gzip xz-utils"

ENV DEV_PACKAGES ${DEV_PACKAGES}

RUN --mount=type=cache,id=dev-apt-cache,sharing=locked,target=/var/cache/apt \
  --mount=type=cache,id=dev-apt-lib,sharing=locked,target=/var/lib/apt \
  apt-get update -qq && \
  apt-get install --no-install-recommends -y ${DEV_PACKAGES} \
  && rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Build gems
FROM build_deps as gems

ARG BUNDLER_VERSION=2.3.26

ENV BUNDLER_VERSION ${BUNDLER_VERSION}

RUN gem install -N bundler -v ${BUNDLER_VERSION}

COPY Gemfile* ./
RUN bundle install && rm -rf vendor/bundle/ruby/*/cache

# Production image
FROM base

ARG NODE_VERSION=18.12.1
ARG PROD_PACKAGES="postgresql-client file vim curl gzip gnupg ca-certificates wget xz-utils"
ARG SERVER_COMMAND="bin/rails fly:server"

ENV PORT 8080
ENV PROD_PACKAGES=${PROD_PACKAGES}
ENV SERVER_COMMAND ${SERVER_COMMAND}

RUN groupadd --gid 1000 node \
  && useradd --uid 1000 --gid node --shell /bin/bash --create-home node

RUN --mount=type=cache,id=prod-apt-cache,sharing=locked,target=/var/cache/apt \
  --mount=type=cache,id=prod-apt-lib,sharing=locked,target=/var/lib/apt \
  apt-get update -qq && \
  apt-get install --no-install-recommends -y \
  ${PROD_PACKAGES} \
  && rm -rf /var/lib/apt/lists /var/cache/apt/archives

RUN set -ex \
  && for key in \
  4ED778F539E3634C779C87C6D7062848A1AB005C \
  141F07595B7B3FFE74309A937405533BE57C7D57 \
  74F12602B6F1C4E913FAA37AD3A89613643B6201 \
  61FC681DFB92A079F1685E77973F295594EC4689 \
  8FCCA13FEF1D0C2E91008E09770F7A9A5AE15600 \
  C4F0DFFF4E8C1A8236409D08E73BC641CC11F4C8 \
  890C08DB8579162FEE0DF9DB8BEAB4DFCF555EF4 \
  C82FA3AE1CBEDC6BE46B9360C43CEC45C17AB93C \
  108F52B48DB57BB0CC439B2997B01419BD92F80A \
  ; do \
  gpg --keyserver hkps://keys.openpgp.org --recv-keys "$key"; \
  done

RUN ARCH= && dpkgArch="$(dpkg --print-architecture)" \
  && case "${dpkgArch##*-}" in \
  amd64) ARCH='x64';; \
  ppc64el) ARCH='ppc64le';; \
  s390x) ARCH='s390x';; \
  arm64) ARCH='arm64';; \
  armhf) ARCH='armv7l';; \
  i386) ARCH='x86';; \
  *) echo "unsupported architecture"; exit 1 ;; \
  esac \
  && set -x \
  && curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-$ARCH.tar.xz" \
  && curl -fsSLO --compressed "https://nodejs.org/dist/v$NODE_VERSION/SHASUMS256.txt.asc" \
  && gpg --batch --decrypt --output SHASUMS256.txt SHASUMS256.txt.asc \
  && grep " node-v$NODE_VERSION-linux-$ARCH.tar.xz\$" SHASUMS256.txt | sha256sum -c - \
  && tar -xJf "node-v$NODE_VERSION-linux-$ARCH.tar.xz" -C /usr/local --strip-components=1 --no-same-owner \
  && rm "node-v$NODE_VERSION-linux-$ARCH.tar.xz" SHASUMS256.txt.asc SHASUMS256.txt \
  && apt-get purge -y --auto-remove xz-utils \
  && ln -s /usr/local/bin/node /usr/local/bin/nodejs

RUN npm install -g mjml

COPY --from=gems /app /app

ENV SECRET_KEY_BASE 1

COPY . .

RUN bin/rails fly:build

EXPOSE ${PORT}

CMD ${SERVER_COMMAND}

fly:build is just an alias to assets:precompile

casperisfine commented 1 year ago

Ok, so I suspect the issue might come from the fact that you compile the gems in a container, then import in your image.

I assume build-essential upgrade the libc or something.

Just for debugging purposes can you try running that same apt-get install in the main image? If that fix the issue it will confirm that it's the problem. And then the proper fix might be to upgrade only the libc package to make sure it match the one used for compilation.

gabrielrinaldi commented 1 year ago

Removing the caching does solve the problems (everything in one image), but adding build-essential to the final image does not make a difference.

I think it is fine to close this as a bootsnap related issue, the issue seems to be ubuntu-latest becoming 22 vs 20. It just coincided for me that both happened at the same time (bootsnap upgrade and the ubuntu-latest).

slondr commented 1 year ago

Can this issue be re-opened? gabrielrinaldi's problem seems unrelated to mine — I am not using containers or macOS.

casperisfine commented 1 year ago

The issue is likely to be similar nonetheless. Please try to flush your bundler cache (or just uninstall bootsnap and re-bundle).

If the issue persists I'll re-open, but I'll need more info to investigate.

tcarac commented 1 year ago

I'm observing a similar issue after having upgraded to ruby 3.2

/home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/setup.rb:5:in<top (required)>' /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap.rb:100:in default_setup' /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap.rb:57:insetup' /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache.rb:16:in setup' /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache.rb:16:inrequire_relative' /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/compile_cache/iseq.rb:3:in <top (required)>' /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:inrequire' /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in require' LoadError: /lib/x86_64-linux-gnu/libc.so.6: versionGLIBC_2.33' not found (required by /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so) - /home/deployer/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/bootsnap.so`

slondr commented 1 year ago

@casperisfine Uninstalling bootsnap, re-bundling, then reinstalling bootsnap did not fix the issue.

What other info do you need from me?

casperisfine commented 1 year ago

What other info do you need from me?

At this point, I'm at a loss as to what it might be. If you could repro this in isolation via a Dockerfile that would help a lot.

In the meantime I'll ask some colleagues if that rings a bell.

XrXr commented 1 year ago

Does ruby -e 'require "bootsnap"; require "bootsnap/bootsnap"' give you the same error? If so, something is off with the native extension. Maybe there are some hints in the output from MAKE='make V=1' gem install bootsnap --verbose --version 1.15.0 and ruby -rrbconfig -e 'pp RbConfig::CONFIG'.

joemsak commented 1 year ago

I had the same issue with my docker container and ruby 3.2.1

But I confirmed for me it was not bootsnap, by commenting out the line that required it, the same error just moved on to another gem

So for me the problem was because I had bundle path set to vendor/bundle and had ran "bundle update" from my host machine. Removing vendor/bundle and forcing our setup scripts to run bundle install inside the container solved it for me.

They key point I found in my google search was some linux forums result for the same issue on another library, someone had said the library had compiled with a different version of glibc. That made me realize I had installed my gems against my host machine's glibc and not the container's

I hope this helps someone else and sorry if it is not relevant to OP's situation

casperisfine commented 1 year ago

Thanks for the info. I'll close then.

eggplants commented 1 year ago

I faced this problem when I try to run rspec test inside of the container on GitHub Actions...

<internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require': /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/bootsnap.so) - /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/bootsnap.so (LoadError)
    from <internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
    from /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/compile_cache/iseq.rb:3:in `<top (required)>'
    from /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/compile_cache.rb:16:in `require_relative'
    from /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/compile_cache.rb:16:in `setup'
    from /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap.rb:57:in `setup'
    from /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap.rb:100:in `default_setup'
    from /app/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.16.0/lib/bootsnap/setup.rb:5:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from <internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from /app/config/boot.rb:5:in `<top (required)>'
    from /app/config/application.rb:1:in `require_relative'
    from /app/config/application.rb:1:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from <internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
    from /app/vendor/bundle/ruby/3.0.0/gems/spring-4.1.1/lib/spring/application.rb:92:in `preload'
    from /app/vendor/bundle/ruby/3.0.0/gems/spring-4.1.1/lib/spring/application.rb:166:in `serve'
    from /app/vendor/bundle/ruby/3.0.0/gems/spring-4.1.1/lib/spring/application.rb:148:in `block in run'
    from /app/vendor/bundle/ruby/3.0.0/gems/spring-4.1.1/lib/spring/application.rb:142:in `loop'
    from /app/vendor/bundle/ruby/3.0.0/gems/spring-4.1.1/lib/spring/application.rb:142:in `run'
    from /app/vendor/bundle/ruby/3.0.0/gems/spring-4.1.1/lib/spring/application/boot.rb:19:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from -e:1:in `<main>'
Error: Process completed with exit code 1.
eggplants commented 1 year ago

Anyway, this problem was resolved with:

diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml
index b872ba633..5c0dfd1b4 100644
--- a/.github/workflows/test.yaml
+++ b/.github/workflows/test.yaml
@@ -24,12 +24,14 @@ jobs:
       - uses: actions/checkout@v3

       # Install Gems with Bundler
-      - uses: ruby/setup-ruby@v1
-        with:
-          bundler-cache: true
-      - run: |
-          bundle config path vendor/bundle
-          bundle install --jobs 4
+      # https://github.com/Shopify/bootsnap/issues/429#issuecomment-1491988620
+      # - uses: ruby/setup-ruby@v1
+      #   with:
+      #     bundler-cache: true
+      # - run: |
+      #     bundle config path vendor/bundle
+      #     bundle install --jobs 4

       # Install Docker Layers
       - id: buildx
pboling commented 1 day ago

I got this error when my CI (BitBucket Pipelines) was attempting to run my specs.

An error occurred while loading rails_helper.
Failure/Error: require File.expand_path("../config/environment", __dir__)
LoadError:
  /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /opt/atlassian/pipelines/agent/build/vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/bootsnap.so) - /opt/atlassian/pipelines/agent/build/vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/bootsnap.so
# /usr/local/lib/ruby/3.3.0/bundled_gems.rb:75:in `require'
# /usr/local/lib/ruby/3.3.0/bundled_gems.rb:75:in `block (2 levels) in replace_require'
# ./vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
# ./vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/compile_cache/iseq.rb:3:in `<top (required)>'
# ./vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/compile_cache.rb:15:in `require_relative'
# ./vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/compile_cache.rb:15:in `setup'
# ./vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap.rb:68:in `setup'
# ./vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap.rb:112:in `default_setup'
# ./vendor/bundle/ruby/3.3.0/gems/bootsnap-1.18.4/lib/bootsnap/setup.rb:5:in `<top (required)>'
# /usr/local/lib/ruby/3.3.0/bundled_gems.rb:75:in `require'

I am on the latest versions of Ruby v3.3.5, Rails v7.2.1.2, Rubygems v3.5.22, bundler v2.5.22.

I thought perhaps it was an issue of missing a platform specification in my Gemfile.lock, and found I only had one: ruby.

So I bundle lock --add-platform ... a bunch of platforms, mimicking what I've had to do for other projects running on the same environments, ending up with:

PLATFORMS
  arm64-darwin
  arm64-darwin-22
  arm64-darwin-23
  ruby
  x86_64-darwin
  x86_64-linux

But the error remained exactly the same. My build script already installs the usual suspects, and makes the usual mods:

            export DEBIAN_FRONTEND=noninteractive
            apt-get update < /dev/null > /dev/null
            apt-get install -qq \
              build-essential \
              libsnappy-dev \
              memcached libmemcached-tools \
              openssl \
              ruby-dev < /dev/null > /dev/null
            gem update --silent --system
            bundle config set deployment 'true'
            bundle config set --local without 'debase development cucumber'
            bundle config set path 'vendor/bundle'
            chmod +t -R vendor/bundle
            chmod o-w -R vendor/bundle
            bundle install --quiet --jobs 4 --retry 3