rails / spring

Rails application preloader
MIT License
2.81k stars 341 forks source link

Fix deadlock issue from threads autoloading and requiring the same dependency #688

Open evanlok opened 1 year ago

evanlok commented 1 year ago

Spring occasionally crashes on boot due to deadlock errors that occur from calling require and autoload of Bundler::SharedHelpers in separate threads.

Ruby bug: https://bugs.ruby-lang.org/issues/15598

Error stack trace:

/usr/local/bundle/gems/bundler-2.3.24/lib/bundler/shared_helpers.rb:12:in `<module:Bundler>': No live threads left. Deadlock?
2 threads, 2 sleeps current:0x000055ccc7d6b950 main thread:0x000055ccc7941f00
* #<Thread:0x000055ccc796ed78 sleep_forever>
   rb_thread_t:0x000055ccc7941f00 native:0x00007fe7557b4f00 int:0
   /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/shared_helpers.rb:12:in `<module:Bundler>'
   /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/shared_helpers.rb:11:in `<top (required)>'
   /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/setup.rb:3:in `require_relative'
   /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/setup.rb:3:in `<top (required)>'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/commands.rb:37:in `<module:Spring>'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/commands.rb:4:in `<top (required)>'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:87:in `preload'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:166:in `serve'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:148:in `block in run'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:142:in `loop'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:142:in `run'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application/boot.rb:19:in `<top (required)>'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   -e:1:in `<main>'
* #<Thread:0x000055ccc7c52620 /usr/local/bundle/gems/spring-4.1.0/lib/spring/failsafe_thread.rb:6 sleep_forever>
   rb_thread_t:0x000055ccc7d6b950 native:0x00007fe751eb5700 int:0
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
   /usr/local/bundle/gems/bundler-2.3.24/lib/bundler.rb:462:in `default_gemfile'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/configuration.rb:14:in `gemfile'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/configuration.rb:63:in `find_project_root'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/configuration.rb:53:in `project_root_path'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/configuration.rb:44:in `application_root_path'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/env.rb:20:in `root'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/env.rb:67:in `app_name'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:36:in `app_name'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/application/boot.rb:15:in `block in <top (required)>'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/process_title_updater.rb:44:in `value'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/process_title_updater.rb:14:in `block in run'
   /usr/local/bundle/gems/spring-4.1.0/lib/spring/failsafe_thread.rb:8:in `block in failsafe_thread'
 (fatal)
    from /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/shared_helpers.rb:11:in `<top (required)>'
    from /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/setup.rb:3:in `require_relative'
    from /usr/local/bundle/gems/bundler-2.3.24/lib/bundler/setup.rb:3:in `<top (required)>'
    from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
    from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
    from /usr/local/bundle/gems/spring-4.1.0/lib/spring/commands.rb:37:in `<module:Spring>'
    from /usr/local/bundle/gems/spring-4.1.0/lib/spring/commands.rb:4:in `<top (required)>'
    from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
    from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
    from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:87:in `preload'
    from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:166:in `serve'
    from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:148:in `block in run'
    from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:142:in `loop'
    from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application.rb:142:in `run'
    from /usr/local/bundle/gems/spring-4.1.0/lib/spring/application/boot.rb:19:in `<top (required)>'
    from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
    from /usr/local/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:85:in `require'
    from -e:1:in `<main>'
rafaelfranca commented 1 year ago

How do we reproduce this deadlock? I could not reproduce it.

evanlok commented 1 year ago

It's difficult to reproduce reliably since it's a race condition between threads. You may be able to reproduce it by writing a script that creates 2 threads: 1 to require shared_helpers and 1 to autoload shared_helpers then invoke that script several times in a loop. We saw this issue occur somewhat often in our CI builds (at least once per day) until this fix was applied. Note that we were running on Ruby 2.7.2. The root cause may have been addressed in newer versions of Ruby although the Ruby bug does not indicate any resolution.