search

rubygems / rubygems

Library packaging and distribution for Ruby.
https://rubygems.org/
Other
3.69k stars 1.75k forks source link

Inconsistent platforms inserted into Gemfile.lock #7710

Open runephilosof-abtion opened 5 months ago

runephilosof-abtion commented 5 months ago

Describe the problem as clearly as you can

When you generate Gemfile.lock from scratch with nokogiri in it, bundler adds four linux architectures and two darwin architectures.

PLATFORMS
  aarch64-linux
  arm-linux
  arm64-darwin
  x86-linux
  x86_64-darwin
  x86_64-linux

If you already have Gemfile.lock with one of the architectures it does not add any new architectures.

If you generate Gemfile.lock from an almost empty Gemfile, you get something like

PLATFORMS
  ruby
  x86_64-linux

There are two differences. ruby is inserted in the last example. And too many architectures are inserted in the first one.

Did you try upgrading rubygems & bundler?

rubygems 3.5.11 bundler 2.5.11

Post steps to reproduce the problem

docker run --rm -it ruby:3.3.1 bash
gem update --system
bundle init
bundle add nokogiri
cat Gemfile.lock
#PLATFORMS
#  aarch64-linux
#  arm-linux
#  arm64-darwin
#  x86-linux
#  x86_64-darwin
#  x86_64-linux
docker run --rm -it ruby:3.3.1 bash
gem update --system
bundle init
bundle # Creates Gemfile.lock with PLATFORMS [ruby, x86_64-linux]
bundle add nokogiri
cat Gemfile.lock
#PLATFORMS
#  ruby
#  x86_64-linux

Which command did you run?

What were you expecting to happen?

I expect the two lock files to be identical.

What actually happened?

There are two differences. ruby is inserted in the last example. And too many architectures are inserted in the first one.

If not included with the output of your command, run bundle env and paste the output below

runephilosof-abtion commented 5 months ago

Also, when ruby is not in platforms, it seems like you are missing the platform independent part of nokogiri

root@1196f9e7a296:/# bundle env
## Environment

Bundler 2.5.11 Platforms ruby, x86_64-linux Ruby 3.3.1p55 (2024-04-23 revision c56cd86388092faec079981f779f140717020d58) [x86_64-linux] Full Path /usr/local/bin/ruby Config Dir /usr/local/etc RubyGems 3.5.11 Gem Home /usr/local/bundle Gem Path /root/.local/share/gem/ruby/3.3.0:/usr/local/lib/ruby/gems/3.3.0:/usr/local/bundle User Home /root User Path /root/.local/share/gem/ruby/3.3.0 Bin Dir /usr/local/bundle/bin Tools
Git 2.39.2 RVM not installed rbenv not installed chruby not installed


## Bundler Build Metadata

Built At 2024-05-28 Git SHA 4afb2d450a Released Version true


## Bundler settings

app_config Set via BUNDLE_APP_CONFIG: "/usr/local/bundle" silence_root_warning Set via BUNDLE_SILENCE_ROOT_WARNING: true


## Gemfile

### Gemfile

```ruby
# frozen_string_literal: true

source "https://rubygems.org"

# gem "rails"

gem "nokogiri", "~> 1.16"

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    nokogiri (1.16.5-aarch64-linux)
      racc (~> 1.4)
    nokogiri (1.16.5-arm-linux)
      racc (~> 1.4)
    nokogiri (1.16.5-arm64-darwin)
      racc (~> 1.4)
    nokogiri (1.16.5-x86-linux)
      racc (~> 1.4)
    nokogiri (1.16.5-x86_64-darwin)
      racc (~> 1.4)
    nokogiri (1.16.5-x86_64-linux)
      racc (~> 1.4)
    racc (1.8.0)

PLATFORMS
  aarch64-linux
  arm-linux
  arm64-darwin
  x86-linux
  x86_64-darwin
  x86_64-linux

DEPENDENCIES
  nokogiri (~> 1.16)

BUNDLED WITH
   2.5.11

Adding the `ruby` platform adds `nokogiri` and `mini_portile2`

root@1196f9e7a296:/# bundle env

Environment

Bundler       2.5.11
  Platforms   ruby, x86_64-linux
Ruby          3.3.1p55 (2024-04-23 revision c56cd86388092faec079981f779f140717020d58) [x86_64-linux]
  Full Path   /usr/local/bin/ruby
  Config Dir  /usr/local/etc
RubyGems      3.5.11
  Gem Home    /usr/local/bundle
  Gem Path    /root/.local/share/gem/ruby/3.3.0:/usr/local/lib/ruby/gems/3.3.0:/usr/local/bundle
  User Home   /root
  User Path   /root/.local/share/gem/ruby/3.3.0
  Bin Dir     /usr/local/bundle/bin
Tools         
  Git         2.39.2
  RVM         not installed
  rbenv       not installed
  chruby      not installed

Bundler Build Metadata

Built At          2024-05-28
Git SHA           4afb2d450a
Released Version  true

Bundler settings

app_config
  Set via BUNDLE_APP_CONFIG: "/usr/local/bundle"
silence_root_warning
  Set via BUNDLE_SILENCE_ROOT_WARNING: true

Gemfile

Gemfile

# frozen_string_literal: true

source "https://rubygems.org"

# gem "rails"

gem "nokogiri", "~> 1.16"

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    nokogiri (1.16.5-aarch64-linux)
      racc (~> 1.4)
    nokogiri (1.16.5-arm-linux)
      racc (~> 1.4)
    nokogiri (1.16.5-arm64-darwin)
      racc (~> 1.4)
    nokogiri (1.16.5-x86-linux)
      racc (~> 1.4)
    nokogiri (1.16.5-x86_64-darwin)
      racc (~> 1.4)
    nokogiri (1.16.5-x86_64-linux)
      racc (~> 1.4)
    racc (1.8.0)

PLATFORMS
  aarch64-linux
  arm-linux
  arm64-darwin
  x86-linux
  x86_64-darwin
  x86_64-linux

DEPENDENCIES
  nokogiri (~> 1.16)

BUNDLED WITH
   2.5.11
deivid-rodriguez commented 5 months ago

This is all expected.

The reason for adding all available plafforms to a fresh generated lockfile is a recent change to make lockfiles more useful in more contexts by default. See https://github.com/rubygems/rubygems/issues/5338 for example. Platforms are only added when specific variants are available for the gems that you are using. If you have an empty Gemfile, then no specific platforms are added (except for the current one). We could probably leave only Ruby in there in that case (actually that's what I expected Bundler to do).

For existing lockfiles, I think it's too surprising that Bundler suddenly starts adding new plaftorms for no reason, that's why that change was only implemented for fresh lockfiles. You can add platforms manually with bundle lock --add-platform to existing lockfiles if you need to.

I will document all this to include the above information.

runephilosof-abtion commented 5 months ago

Did you notice that

docker run --rm -it ruby:3.3.1 bash
gem update --system
bundle init
bundle add nokogiri

Results in a Gemfile without the ruby platform and that this means the dependency mini_portile is missing.

    mini_portile2 (2.8.7)
    nokogiri (1.16.6)
      mini_portile2 (~> 2.8.2)
      racc (~> 1.4)

:point_up: that must be a bug.

Also, I think the Gemfile.lock should only hold version data. The required platforms should be defined in the Gemfile.

deivid-rodriguez commented 5 months ago

Yeah, more than a bug it's a gotcha of the solution that I came up with in https://github.com/rubygems/rubygems/pull/5700 in order to support "universal lockfiles". The solution was to resolve using the current platform, and then add additional platforms for which the resolution found is valid. In this case, the "ruby" platform would introduce an extra dependency as you observed, which not present in the initial resolution, so we can't really add it since it may result in an invalid resolution. Imagine, for example, that there was a top-level dependency on "mini_portile2 < 2.8".

Also, I think the Gemfile.lock should only hold version data. The required platforms should be defined in the Gemfile.

I guess you're thinking of something like https://github.com/rubygems/rubygems/pull/7172. I wouldn't be opposed to that, but we can't just remove lockfile platforms since they are needed for resolution, given that in the real world, gems with the same name and version may still bring different dependencies depending on the platform.

jeromedalbert commented 2 weeks ago

From my understanding of this thread, it seems like the remaining action items are:

deivid-rodriguez commented 2 weeks ago

Yeah, also not sure about the gotcha, but I'd definitely like to document that all platforms are added for new lockfiles. Not sure what the best place is, maybe bundle lock man page?

jpriollaud commented 1 week ago

It seems these platform changes are causing bitbucket pipelines (docker) rspec tests to fail because the wrong dependencies are being installed.

deivid-rodriguez commented 1 week ago

Could you please elaborate, ideally in a separate issue including reproduction steps?