search

httprb / http

HTTP (The Gem! a.k.a. http.rb) - a fast Ruby HTTP client with a chainable API, streaming support, and timeouts
MIT License
3.01k stars 321 forks source link

Requests to an IPv6 address result in: HTTP::ConnectionError (failed to connect: getaddrinfo: Name or service not known) #730

Closed jeraki closed 1 year ago

jeraki commented 2 years ago

I'm trying to use http.rb to make GET requests to IPv6 addresses and always get the following error:

HTTP::ConnectionError (failed to connect: getaddrinfo: Name or service not known)

I've tried with and without brackets around the IP address and the results are the same. (I'd expect the version without brackets to fail, so that by itself isn't a problem. More detail on why I'm even trying a bracket-less version below.)

Interestingly, making a request to the exact same URL with httparty succeeds when using the bracketed form.

Here is a demonstration of what I see using the IPv6 address for example.com (2606:2800:220:1:248:1893:25c8:1946):

http.rb without brackets:

irb(main):177:0> HTTP.get("http://2606:2800:220:1:248:1893:25c8:1946/foo?bar=baz")
Traceback (most recent call last):
       15: from script/rails:6:in `<main>'
       14: from script/rails:6:in `require'
       13: from /redacted/vendor/bundle/ruby/2.6.0/gems/railties-3.2.22.30/lib/rails/commands.rb:41:in `<top (required)>'
       12: from /redacted/vendor/bundle/ruby/2.6.0/gems/railties-3.2.22.30/lib/rails/commands/console.rb:8:in `start'
       11: from /redacted/vendor/bundle/ruby/2.6.0/gems/railties-3.2.22.30/lib/rails/commands/console.rb:47:in `start'
       10: from (irb):177
        9: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/chainable.rb:20:in `get'
        8: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/chainable.rb:77:in `request'
        7: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/client.rb:30:in `request'
        6: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/client.rb:67:in `perform'
        5: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/client.rb:67:in `new'
        4: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/connection.rb:43:in `initialize'
        3: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/timeout/null.rb:21:in `connect'
        2: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/timeout/null.rb:21:in `open'
        1: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/timeout/null.rb:21:in `initialize'
HTTP::ConnectionError (failed to connect: getaddrinfo: Name or service not known)

http.rb with brackets:

irb(main):178:0> HTTP.get("http://[2606:2800:220:1:248:1893:25c8:1946]/foo?bar=baz")
Traceback (most recent call last):
       16: from script/rails:6:in `<main>'
       15: from script/rails:6:in `require'
       14: from /redacted/vendor/bundle/ruby/2.6.0/gems/railties-3.2.22.30/lib/rails/commands.rb:41:in `<top (required)>'
       13: from /redacted/vendor/bundle/ruby/2.6.0/gems/railties-3.2.22.30/lib/rails/commands/console.rb:8:in `start'
       12: from /redacted/vendor/bundle/ruby/2.6.0/gems/railties-3.2.22.30/lib/rails/commands/console.rb:47:in `start'
       11: from (irb):178
       10: from (irb):178:in `rescue in irb_binding'
        9: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/chainable.rb:20:in `get'
        8: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/chainable.rb:77:in `request'
        7: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/client.rb:30:in `request'
        6: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/client.rb:67:in `perform'
        5: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/client.rb:67:in `new'
        4: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/connection.rb:43:in `initialize'
        3: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/timeout/null.rb:21:in `connect'
        2: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/timeout/null.rb:21:in `open'
        1: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/timeout/null.rb:21:in `initialize'
HTTP::ConnectionError (failed to connect: getaddrinfo: Name or service not known)

httparty without brackets:

irb(main):179:0> HTTParty.get("http://2606:2800:220:1:248:1893:25c8:1946/foo?bar=baz")
Traceback (most recent call last):
       16: from script/rails:6:in `<main>'
       15: from script/rails:6:in `require'
       14: from /redacted/vendor/bundle/ruby/2.6.0/gems/railties-3.2.22.30/lib/rails/commands.rb:41:in `<top (required)>'
       13: from /redacted/vendor/bundle/ruby/2.6.0/gems/railties-3.2.22.30/lib/rails/commands/console.rb:8:in `start'
       12: from /redacted/vendor/bundle/ruby/2.6.0/gems/railties-3.2.22.30/lib/rails/commands/console.rb:47:in `start'
       11: from (irb):179
       10: from (irb):179:in `rescue in irb_binding'
        9: from /redacted/vendor/bundle/ruby/2.6.0/gems/httparty-0.16.2/lib/httparty.rb:601:in `get'
        8: from /redacted/vendor/bundle/ruby/2.6.0/gems/httparty-0.16.2/lib/httparty.rb:489:in `get'
        7: from /redacted/vendor/bundle/ruby/2.6.0/gems/httparty-0.16.2/lib/httparty.rb:563:in `perform_request'
        6: from /redacted/vendor/bundle/ruby/2.6.0/gems/httparty-0.16.2/lib/httparty.rb:563:in `new'
        5: from /redacted/vendor/bundle/ruby/2.6.0/gems/httparty-0.16.2/lib/httparty/request.rb:63:in `initialize'
        4: from /redacted/vendor/bundle/ruby/2.6.0/gems/httparty-0.16.2/lib/httparty/request.rb:73:in `path='
        3: from /opt/rubies/ruby-2.6/lib/ruby/2.6.0/uri/common.rb:234:in `parse'
        2: from /opt/rubies/ruby-2.6/lib/ruby/2.6.0/uri/rfc3986_parser.rb:73:in `parse'
        1: from /opt/rubies/ruby-2.6/lib/ruby/2.6.0/uri/rfc3986_parser.rb:67:in `split'
URI::InvalidURIError (bad URI(is not URI?): "http://2606:2800:220:1:248:1893:25c8:1946/foo?bar=baz")

httparty with brackets:

irb(main):180:0> HTTParty.get("http://[2606:2800:220:1:248:1893:25c8:1946]/foo?bar=baz")
=> #<HTTParty::Response:0x7f257239b888 parsed_response="<?xml version=\"1.0\" encoding=\"iso-8859-1\"?>\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\n         \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n<html xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"en\" lang=\"en\">\n\t<head>\n\t\t<title>404 - Not Found</title>\n\t</head>\n\t<body>\n\t\t<h1>404 - Not Found</h1>\n\t</body>\n</html>\n", @response=#<Net::HTTPNotFound 404 Not Found readbody=true>, @headers={"content-type"=>["text/html"], "date"=>["Sat, 19 Nov 2022 01:56:17 GMT"], "server"=>["ECS (dab/DC47)"], "content-length"=>["345"], "connection"=>["close"]}>

I realize I'm using old versions of both HTTP libraries as well as Rails (working with a large corporate codebase that's very outdated), but I've also tried this with http.rb v5 in a Pry session outside of the context of this project, and the same error occurs.

Normally I would just use httparty, because it works, but unfortunately I can't because of a bad interaction between httparty and the newrelic_rpm gem. Apparently Net::HTTP needs brackets removed from IPv6 addresses, so httparty has some code that intentionally removes the brackets before handing off the request to Net::HTTP. At this point, New Relic instrumentation hooks into the request, attempts to re-parse the URL using the addressable gem, and raises an exception because it requires brackets around IPv6 addresses. None of this is http.rb's fault, I'm just mentioning it for the context as to why I'm trying use http.rb in the first place. I wanted to try something that doesn't use Net::HTTP under the hood so that there's no conflict in support for brackets around IPv6 addresses. I see that both http.rb and newrelic_rpm use the addressable gem for URLs, so they should be compatible.

Any idea why I'm getting this error from http.rb? I wasn't able to find documentation or issues that specifically mention IPv6 to see examples of how it's supposed to work or how to troubleshoot.

Thank you for your time.

tarcieri commented 2 years ago

For starters:

1: from /redacted/vendor/bundle/ruby/2.6.0/gems/http-3.3.0/lib/http/timeout/null.rb:21:in `initialize'
HTTP::ConnectionError (failed to connect: getaddrinfo: Name or service not known)

The path shows http-3.3.0, or v3.3.0. The latest release is 5.1.0.

Can you upgrade to the latest release and see if the problem persists?

hvaghani221 commented 2 years ago

I can verify that problem persists with http (5.1.0)

jeraki commented 2 years ago

Yes, I mentioned this in my original post:

I realize I'm using old versions of both HTTP libraries as well as Rails (working with a large corporate codebase that's very outdated), but I've also tried this with http.rb v5 in a Pry session outside of the context of this project, and the same error occurs.

I apologize if that wasn't more clear.

tarcieri commented 2 years ago

The backtrace of the error when using v5.1.0 would be helpful.

FWIW it's probably erroring out here:

https://github.com/httprb/http/blob/f4fb336/lib/http/timeout/null.rb#L21

...in which case it would be helpful to instrument it with what's being passed as host, and figure out what it should be instead, as the ultimate error is:

getaddrinfo: Name or service not known
jeraki commented 2 years ago

I checked out the Git repo and tried to make the request from bundle console to get a stack trace with the latest version:

irb(main):002:0> HTTP.get("http://2606:2800:220:1:248:1893:25c8:1946/foo?bar=baz")

Traceback (most recent call last):
       16: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/cli.rb:31:in `dispatch'
       15: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
       14: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
       13: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
       12: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/cli.rb:509:in `console'
       11: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/cli/console.rb:19:in `run'
       10: from (irb):2
        9: from /redacted/co/http/lib/http/chainable.rb:20:in `get'
        8: from /redacted/co/http/lib/http/chainable.rb:75:in `request'
        7: from /redacted/co/http/lib/http/client.rb:31:in `request'
        6: from /redacted/co/http/lib/http/client.rb:70:in `perform'
        5: from /redacted/co/http/lib/http/client.rb:70:in `new'
        4: from /redacted/co/http/lib/http/connection.rb:42:in `initialize'
        3: from /redacted/co/http/lib/http/timeout/null.rb:21:in `connect'
        2: from /redacted/co/http/lib/http/timeout/null.rb:21:in `open'
        1: from /redacted/co/http/lib/http/timeout/null.rb:21:in `initialize'
HTTP::ConnectionError (failed to connect: getaddrinfo: nodename nor servname provided, or not known)
irb(main):003:0>
irb(main):004:0> HTTP::VERSION
=> "5.1.0"

That's with Ruby 2.6.7 (the main application I work on still uses Ruby 2.6). I can try it with a newer version of Ruby if you think that might matter.

jeraki commented 2 years ago

I tried modifying the connect method in lib/http/timeout/null.rb to this:

def connect(socket_class, host, port, nodelay = false)
  p({
    socket_class: socket_class,
    host: host,
    port: port,
    nodelay: nodelay,
  })
  @socket = socket_class.open(host, port)
  @socket.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1) if nodelay
end

And then when making the request in bundle console I see this just before the exception and stack trace:

{:socket_class=>TCPSocket, :host=>"2606:2800:220:1:248:1893:25c8", :port=>1946, :nodelay=>false}

Looks like it's seeing the last section of the address as the port.

jeraki commented 2 years ago

Forgot to mention, here's the debug output if I put brackets around the IP address. It correctly identifies the host and port, but still results in the same exception:

irb(main):002:0> HTTP.get("http://[2606:2800:220:1:248:1893:25c8:1946]/foo?bar=baz")
{:socket_class=>TCPSocket, :host=>"[2606:2800:220:1:248:1893:25c8:1946]", :port=>80, :nodelay=>false}
Traceback (most recent call last):
       16: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
       15: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
       14: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
       13: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/cli.rb:509:in `console'
       12: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/cli/console.rb:19:in `run'
       11: from (irb):2
       10: from (irb):2:in `rescue in irb_binding'
        9: from /redacted/co/http/lib/http/chainable.rb:20:in `get'
        8: from /redacted/co/http/lib/http/chainable.rb:75:in `request'
        7: from /redacted/co/http/lib/http/client.rb:31:in `request'
        6: from /redacted/co/http/lib/http/client.rb:70:in `perform'
        5: from /redacted/co/http/lib/http/client.rb:70:in `new'
        4: from /redacted/co/http/lib/http/connection.rb:42:in `initialize'
        3: from /redacted/co/http/lib/http/timeout/null.rb:27:in `connect'
        2: from /redacted/co/http/lib/http/timeout/null.rb:27:in `open'
        1: from /redacted/co/http/lib/http/timeout/null.rb:27:in `initialize'
HTTP::ConnectionError (failed to connect: getaddrinfo: nodename nor servname provided, or not known)
tarcieri commented 2 years ago

I've confirmed in this case that URI#host returns [2606:2800:220:1:248:1893:25c8:1946]

>> URI.parse("http://[2606:2800:220:1:248:1893:25c8:1946]/foo?bar=baz").host
=> "[2606:2800:220:1:248:1893:25c8:1946]"

So I'm guessing the solution is we need to strip the outer brackets in this case, if the inner address is a valid IPv6 address?

jeraki commented 2 years ago

I tried changing the connect method to strip the brackets (naively, there's probably a safer way) and that gets further but raises a different exception.

Connect:

def connect(socket_class, host, port, nodelay = false)
  host = host.gsub(/\[|\]/, "")
  @socket = socket_class.open(host, port)
  @socket.setsockopt(Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1) if nodelay
end

Usage:

irb(main):001:0> HTTP.get("http://[2606:2800:220:1:248:1893:25c8:1946]/foo?bar=baz")
Traceback (most recent call last):
       16: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/cli.rb:31:in `dispatch'
       15: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
       14: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
       13: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
       12: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/cli.rb:509:in `console'
       11: from /redacted/.rbenv/versions/2.6.7/lib/ruby/gems/2.6.0/gems/bundler-2.2.33/lib/bundler/cli/console.rb:19:in `run'
       10: from (irb):1
        9: from /redacted/co/http/lib/http/chainable.rb:20:in `get'
        8: from /redacted/co/http/lib/http/chainable.rb:75:in `request'
        7: from /redacted/co/http/lib/http/client.rb:31:in `request'
        6: from /redacted/co/http/lib/http/client.rb:70:in `perform'
        5: from /redacted/co/http/lib/http/client.rb:70:in `new'
        4: from /redacted/co/http/lib/http/connection.rb:42:in `initialize'
        3: from /redacted/co/http/lib/http/timeout/null.rb:22:in `connect'
        2: from /redacted/co/http/lib/http/timeout/null.rb:22:in `open'
        1: from /redacted/co/http/lib/http/timeout/null.rb:22:in `initialize'
HTTP::ConnectionError (failed to connect: No route to host - connect(2) for "2606:2800:220:1:248:1893:25c8:1946" port 80)

This might be an issue with IPv6 support on my local network, though. I notice that I get the same error with curl too:

$ curl -i -v "http://[2606:2800:220:1:248:1893:25c8:1946]/foo?bar=baz"
*   Trying 2606:2800:220:1:248:1893:25c8:1946:80...
* Immediate connect fail for 2606:2800:220:1:248:1893:25c8:1946: No route to host
* Closing connection 0
curl: (7) Couldn't connect to server
tarcieri commented 2 years ago

Stripping the brackets is probably the right solution, although you'd want a regex that matches the outer brackets with an inner capture group for valid IPv6 addresses, which if it matches returns the inner capture group.

If you can test that solution where you have IPv6 connectivity, that'd be great.

Perhaps we could test against ::1 when available?

jeraki commented 2 years ago

I tried the above change to connect by monkey patching the class in a Rails console on the server I usually work from and I was able to successfully make a request to an internal IPv6 address that was previously causing the reported error. So it does look like stripping the brackets is the fix!

hvaghani221 commented 1 year ago

Hi @jeraki @tarcieri is there any update on this?

jeraki commented 1 year ago

@harshit-splunk I didn't have time to work on my PR for a few weeks, but I just pushed a commit to move it forward. You can follow #731 for updates.