ruby-concurrency / concurrent-ruby

Modern concurrency tools including agents, futures, promises, thread pools, supervisors, and more. Inspired by Erlang, Clojure, Scala, Go, Java, JavaScript, and classic concurrency patterns.
https://ruby-concurrency.github.io/concurrent-ruby/
Other
5.68k stars 418 forks source link

Passenger workers hang / segfault with Ruby 2.5 / concurrent-ruby 1.1.6 #871

Open astupka opened 4 years ago

astupka commented 4 years ago

Thank you for all your hard work on the latest 1.1.6 release.

I recently upgraded our app gems and started noticing the passenger workers would become unresponsive (request queue = 100) and not shutdown (normal mem/no CPU usage) after ~12-24 hours. Inspecting the error logs, I found some segfaults that appeared related to the finalization code present in 1.1.6 and a possible MRI bug. Rolling back this single gem to 1.1.5 resolved the issue.

Phusion Passenger 5.0.24
Rails 6.0.0
* Operating system:                Ubuntu 14.04.6 LTS
* Ruby implementation:             ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
* `concurrent-ruby` version:      1.1.6
* `concurrent-ruby-ext` installed: no
* `concurrent-ruby-edge` used:     no

Initial segfault data (let me know if the rest would be helpful)

App 26150 stderr: /opt/production_engine/shared/bundled_gems/ruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/atomic/ruby_thread_local_var.rb:113:
App 26150 stderr: [BUG]
App 26150 stderr: Segmentation fault at 0x0000000000000050
App 26150 stderr:
App 26150 stderr: ruby 2.5.0p0 (2017-12-25 revision 61468) [x86_64-linux]
App 26150 stderr:
App 26150 stderr: -- Control frame information -----------------------------------------------
App 26150 stderr: c:0003
App 26150 stderr: p:----
App 26150 stderr: s:0011
App 26150 stderr: e:000010
App 26150 stderr: CFUNC
App 26150 stderr:  :push
App 26150 stderr:
App 26150 stderr: c:0002
App 26150 stderr: p:0017
App 26150 stderr: s:0006
App 26150 stderr: e:000005
App 26150 stderr: BLOCK
App 26150 stderr:  /opt/production_engine/shared/bundled_gems/ruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/atomic/ruby_thr
App 26150 stderr:  [FINISH]
App 26150 stderr:
App 26150 stderr: c:0001
App 26150 stderr: p:0000
App 26150 stderr: s:0003
App 26150 stderr: E:0022e0
App 26150 stderr: (none)
App 26150 stderr:  [FINISH]
App 26150 stderr:
App 26150 stderr:
App 26150 stderr: -- Ruby level backtrace information ----------------------------------------
App 26150 stderr: /opt/production_engine/shared/bundled_gems/ruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/atomic/ruby_thread_local_var.rb:113:in `block in thread_local_finalizer'
App 26150 stderr: /opt/production_engine/shared/bundled_gems/ruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/atomic/ruby_thread_local_var.rb:113:in `push'
App 26150 stderr:
App 26150 stderr: -- Machine register context ------------------------------------------------
App 26150 stderr:  RIP: 0x00007f3d1c008d96
App 26150 stderr:  RBP: 0x0000000000000000
App 26150 stderr:  RSP: 0x00007ffed1b32bb0
App 26150 stderr:
App 26150 stderr:  RAX: 0x00007f3d187d61f8
App 26150 stderr:  RBX: 0x00007f3d1c0084e4
App 26150 stderr:  RCX: 0x00007f3d1e742ba8
App 26150 stderr:
App 26150 stderr:  RDX: 0x00007f3d1c0084ec
App 26150 stderr:  RDI: 0x0000000000000000
App 26150 stderr:  RSI: 0x00007f3d21a0c8e0
App 26150 stderr:
App 26150 stderr:   R8: 0x00007f3d1c2bc068
App 26150 stderr:   R9: 0x00007f3d1ddd5b20
App 26150 stderr:  R10: 0x0000000000000001
App 26150 stderr:
App 26150 stderr:  R11: 0x00007f3d14a0e199
App 26150 stderr:  R12: 0x00007f3d21a2f7a0
App 26150 stderr:  R13: 0x00007f3d232ac4d0
App 26150 stderr:
App 26150 stderr:  R14: 0x00007f3d1dd06248
App 26150 stderr:  R15: 0x00007f3d21a2f7a0
App 26150 stderr:  EFL: 0x0000000000010287
App 26150 stderr:
App 26150 stderr:
astupka commented 4 years ago

Could be addressed in this PR https://github.com/ruby-concurrency/concurrent-ruby/pull/856

crevete commented 4 years ago

I've encountered the same issue with ruby 2.6.6, rails 5.2.4.3, and only when running unit tests with fork call.

[BUG] Segmentation fault at 0x00007f685c9559d0 ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]

When rolling back this gem to 1.1.5, the Segmentation Fault disappears.

UPDATE: with 1.1.5 the Segmentation Fault may still be there...

pitr-ch commented 4 years ago

856 was merged plese test with master.