http fails on an IPv6 network

[WhyNotHugo]

As previously mentioned downstream here and previously here, http seems to fail on an IPv6 network (even if there is NAT64 for connectivity with IPv4-only hosts).

$ cat test.js 
var http = require('http')
var options = {
  host: 'github.com',
  path: '/'
}
callback = function(response) {
  console.log("Something")
}
http.request(options, callback).end()
$ node test.js 

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: connect ENETUNREACH
    at errnoException (net.js:904:11)
    at connect (net.js:766:19)
    at net.js:845:9
    at asyncCallback (dns.js:68:16)
    at Object.onanswer [as oncomplete] (dns.js:121:9)

For the record, NAT64 solves this just fine:

$ dig +short github.com
192.30.252.128
$ ping6 64:ff9b::192.30.252.128
64 bytes from 64:ff9b::c01e:fc80: icmp_seq=1 ttl=50 time=169 ms
64 bytes from 64:ff9b::c01e:fc80: icmp_seq=2 ttl=50 time=167 ms
^C

Or simply:

$ ping6 github.com
PING github.com(64:ff9b::c01e:fc83) 56 data bytes
64 bytes from 64:ff9b::c01e:fc83: icmp_seq=1 ttl=50 time=188 ms
64 bytes from 64:ff9b::c01e:fc83: icmp_seq=2 ttl=50 time=174 ms
^

$ curl -I github.com
HTTP/1.1 301 Moved Permanently
Content-length: 0
Location: https://github.com/
Connection: close

After looking at the code a bit (/lib/net.js:905), it seems that node works under the assumption that:

1. There's IPv4 local connectivity (Not just NAT64, but that the current host actually has an IPv4 address).
2. The remote host is IPv4-accesible (this is not always true).

Both of these may be true or false, regardless of the other.

While setting an optional attribute is simple, this is the language core, and it would require every single existing application to be modified in order to work on IPv6.

[tjfontaine]

Ok, so the problem here is that we should not be preferring one way or another the family. In the meantime you should (unfortunately) resolve the address before you call net.connect as a workaround.

Our call to uv_getaddrinfo should actually be passing AI_ADDRCONFIG

If hints.ai_flags includes the AI_ADDRCONFIG flag, then IPv4 addresses are returned in the list pointed to by res only if the local system has at least one IPv4 address configured, and IPv6 addresses are only returned if the local system has at least one IPv6 address configured. The loopback address is not considered for this case as valid as a configured address.

If no family is passed to net.connect then no family should be sent to the C++ layer and that flag should be set

We probably also want AI_V4MAPPED

If hint.ai_flags specifies the AI_V4MAPPED flag, and hints.ai_family was specified as AF_INET6, and no matching IPv6 addresses could be found, then return IPv4-mapped IPv6 addresses in the list pointed to by res. If both AI_V4MAPPED and AI_ALL are specified in hints.ai_flags, then return both IPv6 and IPv4-mapped IPv6 addresses in the list pointed to by res. AI_ALL is ignored if AI_V4MAPPED is not also specified.

Do you feel comfortable making a PR for this? or maybe @indutny or ...

[WhyNotHugo]

In the meantime you should (unfortunately) resolve the address before you call net.connect as a workaround.

But it's the http module that's actually doing the call, not the application itself, so applications would need to reimplement it entirely (well, copy-paste most of it, maybe) to do this change.

And it's not really that simple: I'm actually using an applications, which, in turn, use libraries that rely on the http module, so doing this change is non-trivial.

If no family is passed to net.connect then no family should be sent to the C++ layer and that flag should be set

Sounds like it's a lot less complicated that I'd anticipated (I though we might have to check available routes and complicated stuff like that to determine which family we'd use).

Do you feel comfortable making a PR for this?

Not really. I'm familiar with JS, but haven't done anything using nodejs so far. I don't think diving into the core libraries is a good start (I initally found this issue downstream,as a user using popcorntime, as linked above).

[tjfontaine]

Ya, I understand that it's frustrating, but there's not really a good solution for the path going forward.

If no one can get to this soon, it will unfortunately have to come in after 0.12, but it's certainly an issue that needs fixed.

[WhyNotHugo]

Also, a recently discovered, things like npm won't work either since they rely on this. Almost any network-related app will fail because of this issue, so it's not a simple issue of updating client code.

[trevnorris]

Fixed by 430678640 and e643fe4c4.

[WhyNotHugo]

The initial test script still fails on node v0.10.32. Shouldn't it be working already? Or didn't the above fix make it into that release?

[cjihrig]

The fix didn't go into the 0.10.x branch. Try the 0.12 or master branches.

[trymbill]

In trying to send a http request to a server that's only accessible via ipv6 (ISP restrictions), I'm getting ENETUNREACH on v0.12.5. Will this not be fixed until v0.13?

[cjihrig]

I think this is being caused by the dns.ADDRCONFIG flag, assuming your machine doesn't have an IPv6 address configured. Can you compare the results of the following two calls:

dns.lookup('yourIPv6server', {}, function() {});
dns.lookup('yourIPv6server', {hints: dns.ADDRCONFIG}, function() {});

[trymbill]

@cjihrig Thanks for the comment. I get the same results from those two calls (err, address, family):

null 'my:ipv6:address' 6

So it can lookup the address, but when I try to http.request, I'm getting ENETUNREACH. I can reach the server through curl and through Chrome, but not via node.

[cjihrig]

Are you able to provide any additional information? Code, debugging output, etc?

[trymbill]

@cjihrig Sure. I'm running this on Heroku (node v0.12.5):

var options = {
  hostname: process.env.HOMEHOST,
  port: process.env.HOMEPORT
};

http.get(options, function(res) {
  console.log("Got response: " + res.statusCode);
}).on('error', function(e) {
  console.log("Got error: " + e);
});

With these results (with NODE_DEBUG=http set):

2015-07-07T18:50:39.528081+00:00 app[web.1]: HTTP 3: call onSocket 0 0
2015-07-07T18:50:39.535390+00:00 app[web.1]: HTTP 3: outgoing message end.
2015-07-07T18:50:39.738304+00:00 app[web.1]:     at connect (net.js:842:19)
2015-07-07T18:50:39.738298+00:00 app[web.1]: HTTP 3: SOCKET ERROR: connect ENETUNREACH Error: connect ENETUNREACH
2015-07-07T18:50:39.738302+00:00 app[web.1]:     at exports._errnoException (util.js:746:11)
2015-07-07T18:50:39.738697+00:00 app[web.1]: Got error: Error: connect ENETUNREACH
2015-07-07T18:50:39.738306+00:00 app[web.1]:     at GetAddrInfoReqWrap.asyncCallback [as callback] (dns.js:81:16)
2015-07-07T18:50:39.738307+00:00 app[web.1]:     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:99:10)
2015-07-07T18:50:39.738305+00:00 app[web.1]:     at net.js:955:9
2015-07-07T18:50:39.739953+00:00 app[web.1]: HTTP 3: removeSocket [mydomainname]:59985:: destroyed: true
2015-07-07T18:50:39.531751+00:00 app[web.1]: HTTP 3: createConnection [mydomainname]:59985:: { host: '[mydomainname]',
2015-07-07T18:50:39.531755+00:00 app[web.1]:   hostname: '[mydomainname]',
2015-07-07T18:50:39.531757+00:00 app[web.1]:   port: '59985',
2015-07-07T18:50:39.531759+00:00 app[web.1]:   path: null,
2015-07-07T18:50:39.531760+00:00 app[web.1]:   servername: '[mydomainname]' }
2015-07-07T18:50:39.533928+00:00 app[web.1]: HTTP 3: sockets [mydomainname]:59985:: 1
2015-07-07T18:50:39.547178+00:00 app[web.1]: Node app is running at localhost:17010
2015-07-07T18:50:39.739592+00:00 app[web.1]: HTTP 3: CLIENT socket onClose

HOMEHOST / [mydomainname] is a raspberry pi running on my home network that doesn't have a publicly accessible ipv4 address. It does however have a ipv6 address that is accessible from the outside (tested both in Chrome and via curl).

If there's anything else I can supply to help resolve this, let me know.

[cjihrig]

I'd like to take Heroku out of the equation. When you say you can connect via curl, I'm assuming this is not from Heroku (your local machine maybe). Are you able to access your pi using node from that same machine?

By the way, my machine isn't very IPv6 friendly. When I asked you to run two variations of dns.lookup(), I got two different results. I'm also unable to curl google over IPv6.

[WhyNotHugo]

I'd like to take Heroku out of the equation.

I ran this on my desktop:

$ cat test.js 
var http = require('http')
var options = {
    host: 'github.com',
      path: '/'
}
callback = function(response) {
    console.log("Something")
}
http.request(options, callback).end()
$ node test.js 
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: connect ENETUNREACH
    at exports._errnoException (util.js:746:11)
    at connect (net.js:842:19)
    at net.js:955:9
    at GetAddrInfoReqWrap.asyncCallback [as callback] (dns.js:81:16)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:99:10)

The initial issue was never fixed for me, currently on node-v.0.12.5.

$ NODE_DEBUG="http net" node test.js
HTTP 22330: call onSocket 0 0
HTTP 22330: createConnection github.com:80:: { port: 80,
  host: 'github.com',
  path: null,
  servername: 'github.com' }
NET 22330: createConnection [ { port: 80,
    host: 'github.com',
    path: null,
    servername: 'github.com',
    encoding: null } ]
NET 22330: pipe false null
NET 22330: connect: find host github.com
NET 22330: connect: dns options [object Object]
HTTP 22330: sockets github.com:80:: 1
HTTP 22330: outgoing message end.
NET 22330: _read
NET 22330: _read wait for connection
NET 22330: destroy
NET 22330: close
NET 22330: close handle
HTTP 22330: SOCKET ERROR: connect ENETUNREACH Error: connect ENETUNREACH
    at exports._errnoException (util.js:746:11)
    at connect (net.js:842:19)
    at net.js:955:9
    at GetAddrInfoReqWrap.asyncCallback [as callback] (dns.js:81:16)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:99:10)
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: connect ENETUNREACH
    at exports._errnoException (util.js:746:11)
    at connect (net.js:842:19)
    at net.js:955:9
    at GetAddrInfoReqWrap.asyncCallback [as callback] (dns.js:81:16)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:99:10)

[cristobalpalmer]

I'm having trouble with 0.12.7 via docker (following https://hub.docker.com/_/node/ one-liner):

$ sudo docker run -it --rm --name my-running-script -v "$PWD":/usr/src/myapp -w /usr/src/myapp node:0.12.7 node test.js                                                                                         
events.js:85
      throw er; // Unhandled 'error' event
            ^
Error: getaddrinfo ENOTFOUND github.com
    at errnoException (dns.js:44:10)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:94:26)

using the same test.js:

$ cat test.js 
var http = require('http')
var options = {
    host: 'github.com',
      path: '/'
}
callback = function(response) {
    console.log("Something")
}
http.request(options, callback).end()

On a host that can clearly talk to github:

$ HEAD github.com |egrep "^200|^Date"
200 OK
Date: Fri, 25 Sep 2015 21:29:09 GMT

How else might I test to confirm this bug?

[rvagg]

@nodejs/docker, any thoughts here? some networking magic with docker that might be getting in the way

[WhyNotHugo]

@rvagg Docker would seem unrelated. I'm getting the exact same result outside docker (eg: on my laptop, no contanier, vm, anything).

[Starefossen]

@cristobalpalmer @hobarrera from an IPv6 only machine?

[WhyNotHugo]

From an IPv6-only, NAT64-routed machine.

(edited typo)