Segmentation fault when used in parallel threads

[solher]

Hi,

When I try to use the Parallel gem to multi-thread my IO with the Linkedin API, I get a segmentation fault. Sometimes on "net/http.rb:920" and sometimes with "openssl/buffering.rb:326". I put the problem on the Linkedin gem because I don't get the error when I do the same with the Google API. Do you have any idea where the problem could come from ?

Thanks

2014-11-20T10:21:27.010930+00:00 app[web.1]: App 104 stderr: /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:920: [BUG] Segmentation fault at 0x00000000000000 2014-11-20T10:21:27.010936+00:00 app[web.1]: App 104 stderr: ruby 2.1.3p242 (2014-09-19 revision 47630) [x86_64-linux] 2014-11-20T10:21:27.010938+00:00 app[web.1]: App 104 stderr: 2014-11-20T10:21:27.010945+00:00 app[web.1]: App 104 stderr: -- Control frame information ----------------------------------------------- 2014-11-20T10:21:27.010947+00:00 app[web.1]: App 104 stderr: c:0027 p:---- s:0156 e:000155 CFUNC :connect 2014-11-20T10:21:27.010948+00:00 app[web.1]: App 104 stderr: c:0026 p:0008 s:0153 e:000152 BLOCK 2014-11-20T10:21:27.010950+00:00 app[web.1]: App 104 stderr: /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:920 2014-11-20T10:21:27.010952+00:00 app[web.1]: App 104 stderr: c:0025 p:0036 s:0151 E:001c30 BLOCK 2014-11-20T10:21:27.010984+00:00 app[web.1]: App 104 stderr: /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/timeout.rb:91 [FINISH] 2014-11-20T10:21:27.010986+00:00 app[web.1]: App 104 stderr: c:0024 p:---- 2014-11-20T10:21:27.010990+00:00 app[web.1]: App 104 stderr: s:0146 e:000145 CFUNC :call 2014-11-20T10:21:27.010992+00:00 app[web.1]: App 104 stderr: c:0023 2014-11-20T10:21:27.010996+00:00 app[web.1]: App 104 stderr: p:0069 s:0142 E:001178 METHOD /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/timeout.rb:101 2014-11-20T10:21:27.010997+00:00 app[web.1]: App 104 stderr: 2014-11-20T10:21:27.011001+00:00 app[web.1]: App 104 stderr: c:0022 p:0508 2014-11-20T10:21:27.011004+00:00 app[web.1]: App 104 stderr: s:0131 E:001508 METHOD /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:920 2014-11-20T10:21:27.011008+00:00 app[web.1]: App 104 stderr: c:0021 2014-11-20T10:21:27.011011+00:00 app[web.1]: App 104 stderr: p:0007 s:0120 e:000119 METHOD /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:863 2014-11-20T10:21:27.011015+00:00 app[web.1]: App 104 stderr: 2014-11-20T10:21:27.011016+00:00 app[web.1]: App 104 stderr: c:0020 p:0036 2014-11-20T10:21:27.011019+00:00 app[web.1]: App 104 stderr: s:0117 e:000116 METHOD /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:852 2014-11-20T10:21:27.011023+00:00 app[web.1]: App 104 stderr: c:0019 2014-11-20T10:21:27.011026+00:00 app[web.1]: App 104 stderr: p:0017 s:0114 e:000113 METHOD /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1369 2014-11-20T10:21:27.011029+00:00 app[web.1]: App 104 stderr: 2014-11-20T10:21:27.011032+00:00 app[web.1]: App 104 stderr: c:0018 p:0124 2014-11-20T10:21:27.011036+00:00 app[web.1]: App 104 stderr: s:0107 e:000106 METHOD /app/vendor/bundle/ruby/2.1.0/gems/oauth-0.4.7/lib/oauth/consumer.rb:161 2014-11-20T10:21:27.011039+00:00 app[web.1]: App 104 stderr: c:0017 2014-11-20T10:21:27.011043+00:00 app[web.1]: App 104 stderr: p:0018 s:0093 e:000092 METHOD /app/vendor/bundle/ruby/2.1.0/gems/oauth-0.4.7/lib/oauth/tokens/consumer_token.rb:25

[hexgnu]

A segfault is something that is way out of my control.

To cause a segfault involves accessing memory that doesn't technically exist. I think that if you are seeing segfaults while using this gem it's more likely something to do with a misconfigured openssl implementation.

Since the linkedin gem doesn't write any C extensions it's very difficult to cause segfaults or VM crashes.

I'd try posting your issue in the ruby tracker and see what they have to say. Also if you could post more context here I might have some more pointers for fixing.

[solher]

Thanks a lot. It seems also to me that it's more of a global problem with parallel execution in Rails. I'm experiencing quite a lot of problems with multithreading in Ruby (and so, in Rails). Crashes with Postgres, ActiveRecord, some gems... And I'm not even using JRuby or Rubinius.

So about this crash, here is my setup:

• Ruby 2.1.3
• Rails 4.2.0.beta2
• Parallel gem for the multithreading
• I tested it on Heroku and I got the same problem.

Here is the other error I randomly get:

App 1614 stderr: /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/openssl/buffering.rb:326: [BUG] Segmentation fault at 0x0007feb73c45a1 App 1614 stderr: App 1614 stderr: ruby 2.1.3p242 (2014-09-19 revision 47630) [x86_64-darwin14.0] App 1614 stderr: App 1614 stderr: -- Crash Report log information -------------------------------------------- App 1614 stderr: See Crash Report log file under the one of following: App 1614 stderr: * ~/Library/Logs/CrashReporter App 1614 stderr: * /Library/Logs/CrashReporter App 1614 stderr: * ~/Library/Logs/DiagnosticReports App 1614 stderr: * /Library/Logs/DiagnosticReports App 1614 stderr: for more details. App 1614 stderr: App 1614 stderr: -- Control frame information ----------------------------------------------- App 1614 stderr: c:0034 p:---- s:0184 e:000183 CFUNC :syswrite App 1614 stderr: c:0033 p:0164 s:0180 e:000179 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/openssl/buffering.rb:326 App 1614 stderr: c:0032 p:0009 s:0171 e:000170 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/openssl/buffering.rb:344 App 1614 stderr: c:0031 p:0028 s:0167 e:000166 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/protocol.rb:211 App 1614 stderr: c:0030 p:0009 s:0162 e:000161 BLOCK /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/protocol.rb:185 App 1614 stderr: c:0029 p:0027 s:0160 e:000159 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/protocol.rb:202 App 1614 stderr: c:0028 p:0007 App 1614 stderr: s:0156 e:000155 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/protocol.rb:184 App 1614 stderr: c:0027 p:0047 s:0152 e:000151 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http/generic_request.rb:325 App 1614 stderr: c:0026 p:0144 s:0145 e:000144 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http/generic_request.rb:136 App 1614 stderr: c:0025 p:0021 s:0139 e:000138 BLOCK /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1406 [FINISH] App 1614 stderr: c:0024 p:---- s:0137 e:000136 CFUNC :catch App 1614 stderr: c:0023 p:0024 s:0133 e:000132 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1405 App 1614 stderr: c:0022 p:0061 s:0126 e:000125 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1378 App 1614 stderr: c:0021 p:0036 s:0119 e:000118 BLOCK /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1371 App 1614 stderr: c:0020 p:0044 s:0117 e:000116 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:853 App 1614 stderr: c:0019 p:0017 s:0114 e:000113 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1369 App 1614 stderr: c:0018 p:0124 s:0107 e:000106 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/oauth-0.4.7/lib/oauth/consumer.rb:161 App 1614 stderr: c:0017 p:0018 s:0093 e:000092 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/oauth-0.4.7/lib/oauth/tokens/consumer_token.rb:25 App 1614 stderr: c:0016 p:0075 s:0087 e:000086 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/oauth-0.4.7/lib/oauth/tokens/access_token.rb:12 App 1614 stderr: c:0015 p:0017 s:0078 e:000077 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/oauth-0.4.7/lib/oauth/tokens/access_token.rb:27 App 1614 stderr: c:0014 p:0037 s:0073 e:000072 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/linkedin-1.0.0/lib/linked_in/helpers/request.rb:15 App 1614 stderr: c:0013 p:0152 s:0067 e:000065 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/linkedin-1.0.0/lib/linked_in/api/query_helpers.rb:27 App 1614 stderr: c:0012 p:0024 s:0058 e:000057 METHOD App 1614 stderr: /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/linkedin-1.0.0/lib/linked_in/api/people.rb:22

[hundredwatt]

@Solher Looks like the twilio gem had a similar issue: https://github.com/twilio/twilio-ruby/issues/93

We put a global mutex around all calls to the twilio gem https://github.com/twilio/twilio-ruby/issues/93#issuecomment-46553628

[solher]

So the solution is to put linkedin gem calls in Mutex.new.synchronize {} ? But, don't we loose the purpose of multithreading ?

[hexgnu]

The only other thing we could do is mutex around the net/http calls which seem to be the problem. When we merge the faraday changes then we can discuss maybe using other http libraries like typhoeus (or make it configurable) that might not have thread safety issues.

[solher]

Ok thanks a lot !

[hundredwatt]

@Solher You can try the faraday connection now using the unmerged branch:

 gem 'linkedin', github: 'hexgnu/linkedin', branch: 'faraday_connection_updated'

If you do, please let us know how it works in #248

[solher]

It works perfectly ! No more seg fault.