Bundler appears to be trying to install despite `install_if`

alex-tan commented 2 years ago

Describe the problem as clearly as you can

Our Gemfile contains the following:

  install_if -> { RUBY_PLATFORM != "aarch64-linux" } do
    gem "sorbet", "0.5.10337", require: false
    gem "spoom"
    gem "tapioca", "0.10.0", require: false
  end

The lockfile contains:

PLATFORMS
  arm64-darwin-21
  x86_64-linux

This works fine except in aarch64-linux where you get this error:

Bundler could not find compatible versions for gem "sorbet":
  In Gemfile:
    sorbet (= 0.5.10337)

    tapioca (= 0.10.0) was resolved to 0.10.0, which depends on
      spoom (>= 1.1.11, ~> 1.1.0) was resolved to 1.1.12, which depends on
        sorbet (>= 0.5.9204)

Could not find gem 'sorbet-static (= 0.5.10337)' with platform 'ruby', which is required by gem 'sorbet (= 0.5.10337)', in rubygems repository https://rubygems.org/ or installed locally.

The source contains the following gems matching 'sorbet-static (= 0.5.10337)':
  * sorbet-static-0.5.10337-java
  * sorbet-static-0.5.10337-universal-darwin-14
  * sorbet-static-0.5.10337-universal-darwin-15
  * sorbet-static-0.5.10337-universal-darwin-16
  * sorbet-static-0.5.10337-universal-darwin-17
  * sorbet-static-0.5.10337-universal-darwin-18
  * sorbet-static-0.5.10337-universal-darwin-19
  * sorbet-static-0.5.10337-universal-darwin-20
  * sorbet-static-0.5.10337-universal-darwin-21
  * sorbet-static-0.5.10337-universal-darwin-22
  * sorbet-static-0.5.10337-x86_64-linux

Did you try upgrading rubygems & bundler?

Yes.

Post steps to reproduce the problem

On an M1 Mac trying bundling with the snippet above.

MacOS 12.5.1 Docker 4.12.0 Bundler 2.3.22

Which command did you run?

bundle

What were you expecting to happen?

It doesn't try to install sorbet.

What actually happened?

It gives an error that sorbet is incompatible.

If not included with the output of your command, run `bundle env` and paste the output below

I redacted the Gemfile as I'm not sure I can share that.

Bundler       2.3.22
  Platforms   ruby, aarch64-linux
Ruby          2.7.4p191 (2021-07-07 revision a21a3b7d23704a01d34bd79d09dc37897e00922a) [aarch64-linux]
  Full Path   /usr/local/rbenv/versions/2.7.4/bin/ruby
  Config Dir  /usr/local/rbenv/versions/2.7.4/etc
RubyGems      3.1.6
  Gem Home    /usr/local/rbenv/versions/2.7.4/lib/ruby/gems/2.7.0
  Gem Path    /root/.gem/ruby/2.7.0:/usr/local/rbenv/versions/2.7.4/lib/ruby/gems/2.7.0
  User Home   /root
  User Path   /root/.gem/ruby/2.7.0
  Bin Dir     /usr/local/rbenv/versions/2.7.4/bin
Tools         
  Git         2.25.1
  RVM         not installed
  rbenv       rbenv 1.2.0-16-gc4395e5
  chruby      not installed

Bundler Build Metadata

Built At          2022-09-07
Git SHA           44fb4c9ef5
Released Version  true

Bundler settings

clean
  Set for your local app (/app/.bundle/config): true
path
  Set via BUNDLE_PATH: "/gems"
rubygems.pkg.github.com
  Set via BUNDLE_RUBYGEMS__PKG__GITHUB__COM: "[redacted]"

deivid-rodriguez commented 2 years ago

Hi @alex-tan!

Unfortunately the way the :install_if option works is that while it prevents installation if its condition does not evaluate to true, it stills needs to resolve all gems together. This is similar to how :group works.

In this case, the sorbet gem is not compatible with the aarch-linux platform, so this can't possibly work.

An alternative for you would be to completely omit these gems from being considered on aarch-linux, like this

if RUBY_PLATFORM != "aarch64-linux"
  gem "sorbet", "0.5.10337", require: false
  gem "spoom"
  gem "tapioca", "0.10.0", require: false
end

That has the gotcha that it will generate lockfile differences depending on the platform where you use it, since it's not possible to share a lockfile between these platforms in this case.

alex-tan commented 2 years ago

@deivid-rodriguez ah interesting. I was hoping I might gain same hidden insight because somehow I was able to get this to work in two other projects and despite this configuration seemingly being exactly the same it fails in the third project.

I’ll close this for now and maybe use two Gemfiles instead.

deivid-rodriguez commented 2 years ago

@deivid-rodriguez ah interesting. I was hoping I might gain same hidden insight because somehow I was able to get this to work in two other projects and despite this configuration seemingly being exactly the same it fails in the third project.

We had some platform matching bugs that most likely made this work by chance on some Bundler versions. I tried Bundler 2.3.7 and Bundler 2.3.22 and none of them worked.

I’ll close this for now and maybe use two Gemfiles instead.

Yeah that should work too.

n-rodriguez commented 1 month ago

Hi there!

Sorry to resurect this issue but I've just stumbled upon the same issue :

# This file was generated by Appraisal

source "https://rubygems.org"

gem "appraisal", git: "https://github.com/thoughtbot/appraisal.git"
gem "combustion"
gem "database_cleaner"
gem "factory_bot"
gem "faker"
gem "generator_spec"
gem "guard-rspec"
gem "pg"
gem "pry"
gem "puma"
gem "rake"
gem "rspec"
gem "rspec-retry"
gem "rubocop"
gem "rubocop-factory_bot"
gem "rubocop-performance"
gem "rubocop-rake"
gem "rubocop-rspec"
gem "simplecov"
gem "rails", "7.2.0"

install_if -> { ENV["DB_ADAPTER"] == "sqlite3" } do
  gem "sqlite3", "~> 1.5.0"
end

install_if -> { ENV["DB_ADAPTER"] == "mysql2" } do
  gem "mysql2"
end

install_if -> { ENV["DB_ADAPTER"] == "trilogy" } do
  gem "activerecord-trilogy-adapter"
end

install_if -> { ENV["DB_ADAPTER"] == "oracle_enhanced" } do
  gem "ruby-oci8"
  gem "activerecord-oracle_enhanced-adapter", git: "https://github.com/rsim/oracle-enhanced.git"
end

install_if -> { ENV["DB_ADAPTER"] == "postgis" } do
  gem "activerecord-postgis-adapter", git: "https://github.com/rgeo/activerecord-postgis-adapter.git"
end

gemspec path: "../"

Unfortunately the way the :install_if option works is that while it prevents installation if its condition does not evaluate to true, it stills needs to resolve all gems together. This is similar to how :group works.

That's what I see in CI :


Run ruby/setup-ruby@v1
  with:
    ruby-version: 3.3
    bundler-cache: true
  env:
    BUNDLE_GEMFILE: /home/runner/work/ajax-datatables-rails/ajax-datatables-rails/gemfiles/rails_7.2.0.gemfile
    ORACLE_COOKIE: sqldev
    ORACLE_FILE: oracle11g/xe/oracle-xe-11.2.0-1.0.x86_64.rpm.zip
    ORACLE_HOME: /u01/app/oracle/product/11.2.0/xe
    ORACLE_SID: XE
    DB_ADAPTER: sqlite3

Fetching https://github.com/rgeo/activerecord-postgis-adapter.git # <- unexpected fetch -> DB_ADAPTER=sqlite3
Fetching https://github.com/rsim/oracle-enhanced.git # <- unexpected fetch -> DB_ADAPTER=sqlite3
Fetching https://github.com/thoughtbot/appraisal.git
Fetching gem metadata from https://rubygems.org/..........
Resolving dependencies...
Could not find compatible versions

Because every version of activerecord-postgis-adapter depends on activerecord ~>
7.2.0.beta2
and every version of activerecord-oracle_enhanced-adapter depends on
activerecord ~> 7.1.0,
every version of activerecord-postgis-adapter is incompatible with
activerecord-oracle_enhanced-adapter >= 0.
So, because rails_7.2.0.gemfile depends on activerecord-oracle_enhanced-adapter
>= 0
  and rails_7.2.0.gemfile depends on activerecord-postgis-adapter >= 0,
  version solving has failed.

it stills needs to resolve all gems together

Is there a way to avoid this?

Thank you!

deivid-rodriguez commented 1 month ago

That's the way install_if works:

Adds the dependency to the bundle.
Resolves gems.
Installs the resolved gem only when the block evaluates to true.

If you want to only add the dependency and resolve gems when a condition evaluates to true, you could instead do:

if ENV["DB_ADAPTER"] == "postgis"
  gem "activerecord-postgis-adapter", git: "https://github.com/rgeo/activerecord-postgis-adapter.git"
end

Of course you won't be able to keep a single consistent lockfile in all environments since the resolved gems may be different with and without the extra dependency, but I think that's what you want?

n-rodriguez commented 1 month ago

Hi @deivid-rodriguez ! Thanks for your quick answer!

Of course you won't be able to keep a single consistent lockfile in all environments since the resolved gems may be different with and without the extra dependency, but I think that's what you want?

Yes. The problem is "how can I implement this in Appraisals file?" because it won't survive to custom Gemfiles regeneration since if ENV["DB_ADAPTER"] == "postgis" will be evaluated on Appraisals file load.

When I implemented install_if in Appraisal gem I thought it would work like this :

Adds the dependency to the bundle when the block evaluates to true.
Resolves gems.
Installs the resolved gem

but actually it's not the case :/

Maybe we need to implement this on Appraisals side :

appraise 'rails_7.2.0' do
  gem 'rails', '7.2.0'

  custom_install_if 'ENV["DB_ADAPTER"] == "sqlite3"' do
    gem 'sqlite3', '~> 1.5.0'
  end

  custom_install_if 'ENV["DB_ADAPTER"] == "mysql2"' do
    gem 'mysql2'
  end

  custom_install_if 'ENV["DB_ADAPTER"] == "trilogy"' do
    gem 'activerecord-trilogy-adapter'
  end

  custom_install_if 'ENV["DB_ADAPTER"] == "oracle_enhanced"' do
    gem 'ruby-oci8'
    gem 'activerecord-oracle_enhanced-adapter', git: 'https://github.com/rsim/oracle-enhanced.git'
  end

  custom_install_if 'ENV["DB_ADAPTER"] == "postgis"' do
    gem 'activerecord-postgis-adapter', git: 'https://github.com/rgeo/activerecord-postgis-adapter.git'
  end
end

that translates to

# This file was generated by Appraisal

source "https://rubygems.org"

gem "appraisal", git: "https://github.com/thoughtbot/appraisal.git"
gem "combustion"
gem "database_cleaner"
gem "factory_bot"
gem "faker"
gem "generator_spec"
gem "guard-rspec"
gem "pg"
gem "pry"
gem "puma"
gem "rake"
gem "rspec"
gem "rspec-retry"
gem "rubocop"
gem "rubocop-factory_bot"
gem "rubocop-performance"
gem "rubocop-rake"
gem "rubocop-rspec"
gem "simplecov"
gem "rails", "7.2.0"

if ENV["DB_ADAPTER"] == "sqlite3" do
  gem "sqlite3", "~> 1.5.0"
end

if ENV["DB_ADAPTER"] == "mysql2" do
  gem "mysql2"
end

if ENV["DB_ADAPTER"] == "trilogy" do
  gem "activerecord-trilogy-adapter"
end

if ENV["DB_ADAPTER"] == "oracle_enhanced" do
  gem "ruby-oci8"
  gem "activerecord-oracle_enhanced-adapter", git: "https://github.com/rsim/oracle-enhanced.git"
end

if ENV["DB_ADAPTER"] == "postgis" do
  gem "activerecord-postgis-adapter", git: "https://github.com/rgeo/activerecord-postgis-adapter.git"
end

gemspec path: "../"

n-rodriguez commented 1 month ago

@deivid-rodriguez I guess implementing this in Bundler is not possible as it would break a lot of things?

Adds the dependency to the bundle when the block evaluates to true.
Resolves gems.
Installs the resolved gem

deivid-rodriguez commented 1 month ago

I feel what you're trying to do is actually what Appraisals achieves, and that you shouldn't need something like install_if in Appraisal. I would instead define each adapter in its own appraise, like

appraise `rails_7.2.0_with_sqlite3` do
  gem 'rails', '7.2.0'
  gem 'sqlite3', '~> 1.5.0'
end

appraise `rails_7.2.0_with_mysql2` do
  gem 'rails', '7.2.0'
  gem 'mysql2'
end

# and so on...

That way you get the benefit of fully locked dependencies, and you don't run into this issue.

I don't think we want to change the behavior of install_if in Bundler.

n-rodriguez commented 1 month ago

@deivid-rodriguez it works, thanks for the hint!

rubygems / rubygems