"ThreadError: can't create Thread: Resource temporarily unavailable" with multithreading

[zozo]

I'm trying to make reel application to load all available cores. But when I start basic example from the wiki and begin stress-test with ab, I'm getting:

$ ./test.rb
E, [2016-07-24T12:36:20.736468 #19445] ERROR -- : Actor crashed!
ThreadError: can't create Thread: Resource temporarily unavailable
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/group/spawner.rb:47:in `initialize'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/group/spawner.rb:47:in `new'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/group/spawner.rb:47:in `instantiate'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/group/spawner.rb:15:in `get'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/actor/system.rb:76:in `get_thread'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-essentials-0.20.5/lib/celluloid/internals/thread_handle.rb:11:in `initialize'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/actor.rb:129:in `new'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/actor.rb:129:in `start'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/cell.rb:43:in `initialize'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid.rb:195:in `new'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid.rb:195:in `new'
        ./test.rb:30:in `block in <main>'
        /home/zozo/.rvm/gems/ruby-2.2.1/bundler/gems/reel-7b3d06f7b895/lib/reel/server.rb:50:in `call'
        /home/zozo/.rvm/gems/ruby-2.2.1/bundler/gems/reel-7b3d06f7b895/lib/reel/server.rb:50:in `handle_connection'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/calls.rb:28:in `public_send'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/calls.rb:28:in `dispatch'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/call/async.rb:7:in `dispatch'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/cell.rb:50:in `block in dispatch'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/cell.rb:76:in `block in task'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/actor.rb:339:in `block in task'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/task.rb:44:in `block in initialize'
        /home/zozo/.rvm/gems/ruby-2.2.1/gems/celluloid-0.17.2/lib/celluloid/task/fibered.rb:14:in `block in create'

That is my test.rb

#!/usr/bin/env ruby

require 'bundler/setup'
Bundler.require(:default)

class MyConnectionHandler
  include Celluloid

  def initialize(connection)
    @connection = connection
    async.run
  rescue Reel::SocketError
    @connection.close
  end

  def run
    @connection.each_request { |req| handle_request(req) }
  end

  def handle_request(request)
    request.respond(200, {}, 'OK!'.freeze)
  end
end

Reel::Server::HTTP.run('0.0.0.0', 3001) do |connection|
  connection.detach
  MyConnectionHandler.new(connection)
end

Gemfile:

source 'https://rubygems.org'

gem 'celluloid', require: 'celluloid/current'
gem 'reel', github: 'celluloid/reel', branch: :master

ab output:

$ ab -c 10 -n 10000 http://192.168.0.22:3001/
This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.0.22 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
apr_socket_recv: Connection reset by peer (104)
Total of 7886 requests completed

I've tested this on ubuntu 14.04.4 (under VirtualBox) and on OS X

Please, tell me the direction to dig into this problem.

[bogdanRada]

I am not a celluloid member, but i have been using celluloid for a long time

And i have a gem which does almost what you need. You can take a look here: https://github.com/bogdanRada/celluloid_pubsub/blob/master/lib/celluloid_pubsub/web_server.rb

As you can see , i think you should first accept the conection in the server and then only detach the connection and send the request ( `connection.request ) to the ConnectionHandler ( Which should be renamed probably to RequestHandler)

It is bad idea to send the connection instance every time you receive a new request, because each of the ConnectionHandler instances will spawn new threads ( beacause of the async.run where they will try each of them to handle same request.

Instead you should create those instances with the request that needs to be handled , avoiding in this case a deadlock, because right now they will try to use same resource.

Hope you understand my point.

[zozo]

@bogdanRada Thanks for you help. Seems like I didn't understand my problem correctly.

Before I had separate process listening different ports and load balancing with nginx, but now I need only one process to simplify integration with other system components. I found the way - create pool of RequestHandler actors and handle requests through this pool. This allow me to load all cores with one process, but performance of this solution is about two times slower. I'm keep digging and hope to find the way to have only one process and good performance.

That is not a reel problem, so I'm closing issue.

[bogdanRada]

From what i know MRI <= 1.9 uses one core per process irrespective of number of threads spawned (due to GIL restrictions).

However Ruby 2.0 has support now for native threads and here are several improvements to Ruby garbage collector in ruby 2.2 as you can see from this post http://stackoverflow.com/questions/28308363/has-the-global-interpreter-lock-gil-been-removed-from-ruby-in-version-2-2 or from release notes https://github.com/ruby/ruby/blob/v2_2_0/NEWS

Also some other improvements have been made to threads in Ruby 2.3 https://github.com/ruby/ruby/blob/ruby_2_3/NEWS

There are though some considerations for Ruby 3 as far as i understood from this article https://medium.com/@franzejr/ruby-3-mri-and-gil-a302577c6634#.mrb1yvwev ( from 2015) to remove GIL .

But until Ruby 3 will be released , i think the best solution would be to use JRuby , which scales the work to use all cores automatically.

If you really need to use all cores then you might consider using JRuby but i am not sure if this is compatible with it. I honestly use JRuby on very rare ocassions.

I am glad though that i could help, if there is anything else that i can help with, let me know. Thanks a lot.