search

celluloid / celluloid-io

UNMAINTAINED: See celluloid/celluloid#779 - Evented sockets for Celluloid actors
https://celluloid.io
MIT License
879 stars 93 forks source link

Resolv::ResolvError: DNS result has no information for ... under load #188

Closed dblock closed 6 years ago

dblock commented 6 years ago

Seeing these when making hundreds of simultaneous connections. They all are using the same load balancer so DNS resolution should always the same. About 2-5% fail with this:

Resolv::ResolvError: DNS result has no information for cerberus-xxxx.lb.slack-msgs.com
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-io-0.17.3/lib/celluloid/io/tcp_socket.rb:102:in `create_socket'
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-io-0.17.3/lib/celluloid/io/tcp_socket.rb:53:in `initialize'
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-io-0.17.3/lib/celluloid/io/socket.rb:39:in `new'
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-io-0.17.3/lib/celluloid/io/socket.rb:39:in `new'
    /app/vendor/bundle/ruby/2.3.0/gems/slack-ruby-client-0.12.0/lib/slack/real_time/concurrency/celluloid.rb:98:in `build_socket'
    /app/vendor/bundle/ruby/2.3.0/gems/slack-ruby-client-0.12.0/lib/slack/real_time/concurrency/celluloid.rb:33:in `run_loop'
    /app/vendor/bundle/ruby/2.3.0/gems/slack-ruby-client-0.12.0/lib/slack/real_time/concurrency/celluloid.rb:28:in `connect!'
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `public_send'
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-0.17.3/lib/celluloid/calls.rb:28:in `dispatch'
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-0.17.3/lib/celluloid/call/sync.rb:16:in `dispatch'
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:50:in `block in dispatch'
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-0.17.3/lib/celluloid/cell.rb:76:in `block in task'
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-0.17.3/lib/celluloid/actor.rb:339:in `block in task'
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-0.17.3/lib/celluloid/task.rb:44:in `block in initialize'
    /app/vendor/bundle/ruby/2.3.0/gems/celluloid-0.17.3/lib/celluloid/task/fibered.rb:14:in `block in create'

via https://github.com/slack-ruby/slack-ruby-client/issues/218

ioquatix commented 6 years ago

That seems odd

dblock commented 6 years ago

It went away 12 hours later, maybe it's a legit DNS resolver problem after all where the round-robin is returning a bad result once in a while. I'm going to close.

HoneyryderChuck commented 6 years ago

The custom Celluloid DNS Resolver misses a lot of the properties that the ruby default resolver has:

Any of these might factor into your announced error rate. If you're using the same domain name, which will resolve to the same IP, I suggest (if possible) that you resolve the address before entering "celluloid land" (using resolv lib) and use the IP directly.

dblock commented 6 years ago

Just for context, in this particular case the slack library has an API call (web) that returns an URL that includes a domain name. I guess I could try resolving that first to improve outcomes, but it seems like it's fine with what it is for now. Thanks everyone.