search

cheald / manticore

Manticore is a JRuby HTTP client built on the Apache HttpClient 4.x components
https://gitlab.com/cheald/manticore
MIT License
54 stars 34 forks source link

ConcurrencyError And NullPointerException seen in parallel requests #80

Closed alkalinecoffee closed 4 years ago

alkalinecoffee commented 5 years ago

I'm working on a simple rack app that makes two requests in parallel and load testing it with wrk. However, I'm often seeing the following error:

2019-01-24 13:33:07 -0500: Rack app error handling request { GET / }
#<ConcurrencyError: Detected invalid array contents due to unsynchronized modifications with concurrent users>
org/jruby/RubyArray.java:2486:in `map'
/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:337:in `execute!'
/Users/jmartin/src/playground/parallel-manticore/app_example.rb:27:in `call'
/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/configuration.rb:225:in `call'
/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/server.rb:658:in `handle_request'
/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/server.rb:472:in `process_client'
/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/server.rb:332:in `block in run'
/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/thread_pool.rb:133:in `block in spawn_thread'

And at times see:

2019-01-24 13:39:27 -0500: Rack app error handling request { GET / }
java.lang.NullPointerException
java.util.concurrent.AbstractExecutorService.submit(java/util/concurrent/AbstractExecutorService.java:132)
java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:453)
org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:314)
org.jruby.RubyMethod.call(org/jruby/RubyMethod.java:115)
org.jruby.RubyMethod$INVOKER$i$call.call(org/jruby/RubyMethod$INVOKER$i$call.gen)
Users.jmartin.$_dot_rvm.gems.jruby_minus_9_dot_1_dot_8_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.invokeOther0:call(Users/jmartin/$_dot_rvm/gems/jruby_minus_9_dot_1_dot_8_dot_0/gems/manticore_minus_0_dot_6_dot_4_minus_java/lib/manticore//Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:337)
Users.jmartin.$_dot_rvm.gems.jruby_minus_9_dot_1_dot_8_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.block in execute!(/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:337)
org.jruby.RubyArray.collect(org/jruby/RubyArray.java:2472)
org.jruby.RubyArray.map(org/jruby/RubyArray.java:2486)
org.jruby.RubyArray$INVOKER$i$0$0$map19.call(org/jruby/RubyArray$INVOKER$i$0$0$map19.gen)
Users.jmartin.$_dot_rvm.gems.jruby_minus_9_dot_1_dot_8_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.invokeOther12:map(Users/jmartin/$_dot_rvm/gems/jruby_minus_9_dot_1_dot_8_dot_0/gems/manticore_minus_0_dot_6_dot_4_minus_java/lib/manticore//Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:337)
Users.jmartin.$_dot_rvm.gems.jruby_minus_9_dot_1_dot_8_dot_0.gems.manticore_minus_0_dot_6_dot_4_minus_java.lib.manticore.client.execute!(/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/manticore-0.6.4-java/lib/manticore/client.rb:337)
config_dot_ru.invokeOther29:execute!(config.ru:25)
config_dot_ru.call(config.ru:25)
Users.jmartin.$_dot_rvm.gems.jruby_minus_9_dot_1_dot_8_dot_0.gems.puma_minus_3_dot_12_dot_0_minus_java.lib.puma.configuration.invokeOther3:call(Users/jmartin/$_dot_rvm/gems/jruby_minus_9_dot_1_dot_8_dot_0/gems/puma_minus_3_dot_12_dot_0_minus_java/lib/puma//Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/configuration.rb:225)
Users.jmartin.$_dot_rvm.gems.jruby_minus_9_dot_1_dot_8_dot_0.gems.puma_minus_3_dot_12_dot_0_minus_java.lib.puma.configuration.call(/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/configuration.rb:225)
Users.jmartin.$_dot_rvm.gems.jruby_minus_9_dot_1_dot_8_dot_0.gems.puma_minus_3_dot_12_dot_0_minus_java.lib.puma.server.invokeOther83:call(Users/jmartin/$_dot_rvm/gems/jruby_minus_9_dot_1_dot_8_dot_0/gems/puma_minus_3_dot_12_dot_0_minus_java/lib/puma//Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/server.rb:658)
Users.jmartin.$_dot_rvm.gems.jruby_minus_9_dot_1_dot_8_dot_0.gems.puma_minus_3_dot_12_dot_0_minus_java.lib.puma.server.handle_request(/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/server.rb:658)
RUBY.process_client(/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/server.rb:472)
RUBY.block in run(/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/server.rb:332)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
org.jruby.RubyProc.call19(org/jruby/RubyProc.java:273)
org.jruby.RubyProc$INVOKER$i$0$0$call19.call(org/jruby/RubyProc$INVOKER$i$0$0$call19.gen)
RUBY.block in spawn_thread(/Users/jmartin/.rvm/gems/jruby-9.1.8.0/gems/puma-3.12.0-java/lib/puma/thread_pool.rb:133)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)
org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)
java.lang.Thread.run(java/lang/Thread.java:748)

I'm seeing this when standing up a rack app with the following sample code:

class TestApp

  CLIENT = Manticore::Client.new(follow_redirects: true, pool_max: 500)

  def call(env)
    response = CLIENT.parallel.get("http://www.yahoo.com")
    response.on_success do |response|
      puts "The length of the Yahoo! homepage is #{response.body.length}"
    end

    response.on_failure do |response|
      puts "http://www.nooooooooooooooo.com/"
    end

    CLIENT.parallel.get("http://bing.com")
      .on_success  {|r| puts r.code     }
      .on_failure  {|e| puts "on noes!" }
      .on_complete { puts "Job's done!" }

    CLIENT.execute!

    [200, {"Content-Type" => "text/html"}, ["Hello World!"]]
  end

end

And using wrk to put some load on this app: wrk -d10 -t10 -c10 http://localhost:9292.

Any suggestions on avoiding this?

puma version 3.12.0
jruby 9.1.8.0
java version 1.8.0_201
manticore 0.6.4
cheald commented 4 years ago

It's been a while since I've been in these issues (so sorry!), but I've reproduced the problem. @async_requests was treated as threadsafe, and isn't. I've swapped it for a Queue, which seems to address the issue.

alkalinecoffee commented 4 years ago

Thanks!!