Closed chrismo closed 7 years ago
Fwiw, this works with BUNDLE_SPECIFIC_PLATFORM=true
, but I'm trying to investigate why its failing without
Yup, its the same bug in LazySpecification#__materialize__
that specific_platform
fixes. I have a potential fix, just need to run the entire test suite to see if it causes any regressions
Yeah, can't see how to fix this without breaking things like ./spec/commands/install_spec.rb:212
(tried the following to no avail)
diff --git a/lib/bundler/installer/parallel_installer.rb b/lib/bundler/installer/parallel_installer.rb
index 70939ceeb..b1a997f3f 100644
--- a/lib/bundler/installer/parallel_installer.rb
+++ b/lib/bundler/installer/parallel_installer.rb
@@ -67,6 +67,10 @@ module Bundler
def all_dependencies
@spec.dependencies
end
+
+ def to_s
+ "#<#{self.class} #{@spec.full_name} (#{state})>"
+ end
end
def self.call(*args)
diff --git a/lib/bundler/lazy_specification.rb b/lib/bundler/lazy_specification.rb
index 7508347c3..fa6501aa9 100644
--- a/lib/bundler/lazy_specification.rb
+++ b/lib/bundler/lazy_specification.rb
@@ -69,12 +69,12 @@ module Bundler
end
def __materialize__
- search_object = Bundler.settings[:specific_platform] || Bundler.settings[:force_ruby_platform] ? self : Dependency.new(name, version)
- @specification = if source.is_a?(Source::Gemspec) && source.gemspec.name == name
- source.gemspec.tap {|s| s.source = source }
- else
- source.specs.search(search_object).last
+ if source.is_a?(Source::Gemspec) && source.gemspec.name == name
+ return @specification = source.gemspec.tap {|s| s.source = source }
end
+ search_object = Bundler.settings[:specific_platform] ? self : Dependency.new(name, version)
+ search = source.specs.search(search_object)
+ @specification = GemHelpers.select_best_platform_match(search, Gem::Platform.new(platform))
end
def respond_to?(*args)
We could maybe change the materialize logic to check if the dependencies are the same or not, but that's getting into murky waters with behavior changes.
I'd like to get @indirect's perspective on what sort of compromises might be acceptable here, given we already have a fix behind a feature flag
Specs:
(only "installs gems for the local platform as first choice"
is failing on master)
diff --git a/lib/bundler/installer/parallel_installer.rb b/lib/bundler/installer/parallel_installer.rb
index 70939ceeb..b1a997f3f 100644
--- a/lib/bundler/installer/parallel_installer.rb
+++ b/lib/bundler/installer/parallel_installer.rb
@@ -67,6 +67,10 @@ module Bundler
def all_dependencies
@spec.dependencies
end
+
+ def to_s
+ "#<#{self.class} #{@spec.full_name} (#{state})>"
+ end
end
def self.call(*args)
diff --git a/spec/commands/install_spec.rb b/spec/commands/install_spec.rb
index 49eee0191..83f47c523 100644
--- a/spec/commands/install_spec.rb
+++ b/spec/commands/install_spec.rb
@@ -252,6 +252,52 @@ RSpec.describe "bundle install with gem sources" do
run "require 'platform_specific' ; puts PLATFORM_SPECIFIC"
expect(out).to eq("1.0.0 MSWIN")
end
+
+ context "with platform-specific dependencies" do
+ it "installs gems for the local platform as first choice" do
+ install_gemfile <<-G
+ source "file://#{gem_repo1}"
+ gem "platform_specific_deps"
+ G
+
+ run "require 'platform_specific_deps' ; puts PLATFORM_SPECIFIC_DEPS"
+ expect(out).to eq("1.0.0 #{Bundler.local_platform}")
+ end
+
+ it "falls back on plain ruby" do
+ simulate_platform "foo-bar-baz"
+ install_gemfile <<-G
+ source "file://#{gem_repo1}"
+ gem "platform_specific_deps"
+ G
+
+ run "require 'platform_specific_deps' ; puts PLATFORM_SPECIFIC_DEPS"
+ expect(out).to eq("1.0.0 RUBY")
+ end
+
+ it "installs gems for java" do
+ simulate_platform "java"
+ install_gemfile <<-G
+ source "file://#{gem_repo1}"
+ gem "platform_specific_deps"
+ G
+
+ run "require 'platform_specific_deps' ; puts PLATFORM_SPECIFIC_DEPS"
+ expect(out).to eq("1.0.0 JAVA")
+ end
+
+ it "installs gems for windows" do
+ simulate_platform mswin
+
+ install_gemfile <<-G
+ source "file://#{gem_repo1}"
+ gem "platform_specific_deps"
+ G
+
+ run "require 'platform_specific_deps' ; puts PLATFORM_SPECIFIC_DEPS"
+ expect(out).to eq("1.0.0 MSWIN")
+ end
+ end
end
describe "doing bundle install foo" do
diff --git a/spec/support/builders.rb b/spec/support/builders.rb
index bda808c0b..8fc780083 100644
--- a/spec/support/builders.rb
+++ b/spec/support/builders.rb
@@ -104,6 +104,35 @@ module Spec
s.write "lib/platform_specific.rb", "PLATFORM_SPECIFIC = '1.0.0 x86-darwin-100'"
end
+ build_gem "platform_specific_deps" do |s|
+ s.platform = Bundler.local_platform
+ s.add_dependency 'activesupport'
+ s.write "lib/platform_specific_deps.rb", "PLATFORM_SPECIFIC_DEPS = '1.0.0 #{Bundler.local_platform}'"
+ end
+
+ build_gem "platform_specific_deps" do |s|
+ s.platform = "java"
+ s.add_dependency 'only_java'
+ s.write "lib/platform_specific_deps.rb", "PLATFORM_SPECIFIC_DEPS = '1.0.0 JAVA'"
+ end
+
+ build_gem "platform_specific_deps" do |s|
+ s.platform = "ruby"
+ s.write "lib/platform_specific_deps.rb", "PLATFORM_SPECIFIC_DEPS = '1.0.0 RUBY'"
+ end
+
+ build_gem "platform_specific_deps" do |s|
+ s.platform = "x86-mswin32"
+ s.add_dependency 'has_prerelease'
+ s.write "lib/platform_specific_deps.rb", "PLATFORM_SPECIFIC_DEPS = '1.0.0 MSWIN'"
+ end
+
+ build_gem "platform_specific_deps" do |s|
+ s.platform = "x86-darwin-100"
+ s.add_dependency 'terranova'
+ s.write "lib/platform_specific_deps.rb", "PLATFORM_SPECIFIC_DEPS = '1.0.0 x86-darwin-100'"
+ end
+
build_gem "only_java", "1.0" do |s|
s.platform = "java"
s.write "lib/only_java.rb", "ONLY_JAVA = '1.0.0 JAVA'"
Wasn't using bundler/inline just bundler and only way to fix it was to use the BUNDLE_SPECIFIC_PLATFORM=true
which I don't think should be a necessary ENV setting.
Here is my Gemfile
source ENV['GEM_SOURCE'] || "https://rubygems.org"
group :test do
gem 'rake', require: false
gem 'puppet', ENV['PUPPET_GEM_VERSION'] || '~> 4.3.2', require: false
gem 'facter', ENV['FACTER_GEM_VERSION'] || '~> 2.4.6', require: false
gem 'rspec', '> 3.2.0', require: false
gem 'rspec-puppet', '>= 2.5.0', require: false
gem 'puppet-lint', '= 2.0.2', require: false
gem 'puppetlabs_spec_helper', require: false
gem 'r10k', require: false
gem 'puppetclassify', require: false
gem 'rspec-puppet-utils', require: false
gem 'rspec-puppet-facts', require: false
gem 'rubocop', '0.43.0', require: false
gem 'simplecov', require: false
gem 'simplecov-console', require: false
gem 'guard-rake', require: false
gem 'listen', '< 3.1.0', require: false
gem 'hiera-eyaml', require: false
gem 'ci_reporter', require: false
gem 'ci_reporter_rspec', require: false
gem 'puppet-lint-absolute_classname-check', require: false
gem 'puppet-lint-empty_string-check', require: false
gem 'puppet-lint-leading_zero-check', require: false
gem 'puppet-lint-roles_and_profiles-check', require: false
#gem 'puppet-lint-spaceship_operator_without_tag-check', require: false
gem 'puppet-lint-undef_in_function-check', require: false
gem 'puppet-lint-unquoted_string-check', require: false
gem 'puppet-lint-variable_contains_upcase', require: false
end
group :system_tests do
gem 'beaker', ENV['BEAKER_VERSION'] || '~> 2.37.0'
gem 'beaker-rspec', ENV['BEAKER_RSPEC_VERSION'] || '~> 5.3.0'
gem 'serverspec', require: false
gem 'beaker-puppet_install_helper', require: false
end
# vim:ft=ruby
Here's the run:
$ bundle install --without system_tests
Fetching gem metadata from https://rubygems.org/.........
Fetching version metadata from https://rubygems.org/..
Fetching dependency metadata from https://rubygems.org/.
Resolving dependencies......................................................................................
Your lockfile was created by an old Bundler that left some things out.
Because of the missing DEPENDENCIES, we can only install gems one at a time, instead of installing 7 at a time.
You can fix this by adding the missing gems to your Gemfile, running bundle install, and then removing the gems from your Gemfile.
The missing gems are:
* CFPropertyList depended upon by facter
* CFPropertyList depended upon by puppet
Installing rake 12.0.0
Installing ansi 1.5.0
Installing ast 2.3.0
Installing builder 3.2.3
Installing rspec-support 3.5.0
Installing diff-lcs 1.3
Installing coderay 1.1.1
Installing colored 1.2
Installing docile 1.1.5
Installing json 1.8.6 with native extensions
Installing multipart-post 2.0.0
Installing fast_gettext 1.1.0
Installing ffi 1.9.17 with native extensions
Installing formatador 0.2.5
Installing locale 2.1.2
Installing text 1.3.1
Installing rb-fsevent 0.9.8
Installing lumberjack 1.0.11
Installing nenv 0.3.0
Installing shellany 0.0.1
Installing method_source 0.8.2
Installing slop 3.6.0
Installing thor 0.19.4
Installing hiera 3.2.2
Installing highline 1.6.21
Installing trollop 2.1.2
Installing hirb 0.7.3
Installing json_pure 1.8.6
Installing log4r 1.1.10
Installing stomp 1.4.3
Installing systemu 2.6.5
Installing metaclass 0.0.4
Installing minitar 0.5.4
Installing multi_json 1.12.1
Installing powerpack 0.1.1
Installing puppet-lint 2.0.2
Installing puppetclassify 0.1.5
Installing rainbow 2.2.1 with native extensions
Installing ruby-progressbar 1.8.1
Installing unicode-display_width 1.1.3
Installing simplecov-html 0.10.0
Using bundler 1.14.4
Installing puppet-syntax 2.3.0
Installing parser 2.4.0.0
Installing ci_reporter 2.0.0
Installing rspec-core 3.5.4
Installing rspec-expectations 3.5.0
Installing rspec-mocks 3.5.0
Installing cri 2.6.1
Installing jgrep 1.4.1
Installing faraday 0.9.2
Installing rb-inotify 0.9.8
Installing gettext 3.2.2
Installing notiffany 0.1.1
Installing pry 0.10.4
Installing hiera-eyaml 2.1.0
Installing mcollective-client 2.9.1
Installing mocha 1.2.1
Installing puppet-lint-absolute_classname-check 0.2.4
Installing puppet-lint-empty_string-check 0.2.2
Installing puppet-lint-leading_zero-check 0.1.1
Installing puppet-lint-roles_and_profiles-check 0.1.1
Installing puppet-lint-undef_in_function-check 0.2.1
Installing puppet-lint-unquoted_string-check 0.3.0
Installing puppet-lint-variable_contains_upcase 1.1.0
Installing simplecov 0.13.0
Installing rubocop 0.43.0
Installing rspec 3.5.0
Installing faraday_middleware 0.10.1
Installing listen 3.0.8
Installing gettext-setup 0.13
Installing simplecov-console 0.4.1
Installing ci_reporter_rspec 1.0.0
Installing rspec-puppet 2.5.0
Installing guard 2.14.1
Installing semantic_puppet 0.1.4
Installing puppetlabs_spec_helper 2.0.1
Installing guard-rake 1.0.0
Installing puppet_forge 2.2.3
Installing r10k 2.5.2
--- ERROR REPORT TEMPLATE -------------------------------------------------------
# Error Report
## Questions
Please fill out answers to these questions, it'll help us figure out
why things are going wrong.
- **What did you do?**
I ran the command `/opt/boxen/rbenv/versions/2.3.3/bin/bundle install --without system_tests`
- **What did you expect to happen?**
I expected Bundler to...
- **What happened instead?**
Instead, what happened was...
- **Have you tried any solutions posted on similar issues in our issue tracker, stack overflow, or google?**
I tried...
- **Have you read our issues document, https://github.com/bundler/bundler/blob/master/doc/contributing/ISSUES.md?**
...
## Backtrace
fatal: No live threads left. Deadlock?
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/worker.rb:43:in pop' /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/worker.rb:43:in
deq'
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/installer/parallel_installer.rb:131:in process_specs' /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/installer/parallel_installer.rb:102:in
call'
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/installer/parallel_installer.rb:77:in call' /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/installer.rb:199:in
install_in_parallel'
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/installer.rb:166:in install' /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/installer.rb:80:in
run'
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/installer.rb:24:in install' /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/cli/install.rb:70:in
run'
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/cli.rb:181:in block in install' /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/settings.rb:99:in
temporary'
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/cli.rb:180:in install' /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/vendor/thor/lib/thor/command.rb:27:in
run'
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in invoke_command' /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/vendor/thor/lib/thor.rb:359:in
dispatch'
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/cli.rb:20:in dispatch' /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/vendor/thor/lib/thor/base.rb:440:in
start'
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/cli.rb:11:in start' /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/exe/bundle:32:in
block in <top (required)>'
/opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/lib/bundler/friendly_errors.rb:121:in with_friendly_errors' /opt/rubies/2.3.3/lib/ruby/gems/2.3.0/gems/bundler-1.14.4/exe/bundle:24:in
<top (required)>'
/opt/boxen/rbenv/versions/2.3.3/bin/bundle:22:in load' /opt/boxen/rbenv/versions/2.3.3/bin/bundle:22:in
## Environment
Bundler 1.14.4 Rubygems 2.6.10 Ruby 2.3.3p222 (2016-11-21 revision 56859) [x86_64-darwin15] GEM_HOME /Users/cdenneen/src/gitlab/gicontrol-puppet/.bundle GEM_PATH Git 2.11.0 Platform x86_64-darwin-15 OpenSSL OpenSSL 1.0.2k 26 Jan 2017
## Bundler settings
jobs Set via BUNDLE_JOBS: "8" Set for the current user (/Users/cdenneen/.bundle/config): "7" path Set for the current user (/Users/cdenneen/.bundle/config): ".bundle" disable_shared_gems Set for your local app (/Users/cdenneen/src/gitlab/gicontrol-puppet/.bundle/config): true Set for the current user (/Users/cdenneen/.bundle/config): "true" gem.test Set for the current user (/Users/cdenneen/.bundle/config): "false" gem.mit Set for the current user (/Users/cdenneen/.bundle/config): "false" gem.coc Set for the current user (/Users/cdenneen/.bundle/config): "false" without Set for your local app (/Users/cdenneen/src/gitlab/gicontrol-puppet/.bundle/config): "system_tests" Set for the current user (/Users/cdenneen/.bundle/config): "system_tests"
--- TEMPLATE END ----------------------------------------------------------------
Unfortunately, an unexpected error occurred, and Bundler cannot continue.
First, try this link to see if there are any existing issue reports for this error:
https://github.com/bundler/bundler/search?q=No+live+threads+left.+Deadlock%3F&type=Issues
If there aren't any reports for this error yet, please create copy and paste the report template above into a new issue. Don't forget to anonymize any private data! The new issue form is located at:
https://github.com/bundler/bundler/issues/new
And oddly there is no Gemfile.lock so not sure why it's talking about my Your lockfile was created by an old Bundler that left some things out.
@segiddins any further info?
The fix is indeed to use BUNDLE_SPECIFIC_PLATFORM (which will be enabled by default in 2.0)
Alternatively, https://github.com/bundler/bundler/pull/5427 should help, if you want to give it a try
@segiddins it will be default in 2.0 of bundler but for now 1.14.5 it will just be a warning rather than failure? Am I reading that correctly?
Adding variables like that to every bundle install
workflow can be majorly disruptive... probably easier if it didn't do that?
specific_platform
will default to true on 2.0, it's optional on 1.x.
That PR should fix your particular case (the facter
gem) and allow it to work without specific_platform
)
@cdenneen sorry this is confusing, let me try to summarize everything all at once:
There is a bug in certain situations, and that circumstances that trigger the bug include both certain gems installed using bundler/inline
and certain API responses from the server that are missing data. The bug in question can be worked around by either setting the specific_platform
setting to true (using an environment variable or running bundle config specific_platform true
), or by running bundle install --full-index
.
Over in PR #5427, we're working on handling the problematic API responses better, and providing users with information about how they can work around the problem without having to come here.
In the long run, we plan to enable the specific_platform
setting as a default—it's arguably the "correct" way to do things, but we will be breaking backwards compatibility when we make that change, so it has to wait for 2.0 to be the default.
Also in the long run, we want to fix the API responses that activate this problem. The data that Bundler needs is supposed to always be included in those responses, so there's a server-side bug we'll need to try to fix as well.
I hope that helps!
As a part of the coverage script I put together for #5332, one of the gems it attempted to install was 'facter' and its latest version '2.4.6'. The exceptions only happen with bundler/inline. No exceptions happen using bundler normally. There are two versions of the exception, one for 1.13.7 and another for 1.14.0.pre.2.
With this script:
1.13.7 results in:
and 1.14.0.pre.2 results in:
I searched the issues for the error message presented by 1.14.0 and found some old issues that had to do with gems with circular dependencies, but I'm pretty sure Bundler's resolver these days detects those AND facter appears to only have a dependency on CFPropertyList.
Since I recreated in 1.13, and it's only with bundler/inline (and this gem so far), this could be a pretty esoteric bug and may not be worth investigating, but wanted to report it anyway.