search

rust-lang / crates.io

The Rust package registry
https://crates.io
Apache License 2.0
2.96k stars 600 forks source link

Network connectivity issues over IPv6 (Hurricane Electric tunnelbroker) #9740

Open aidanharris opened 1 week ago

aidanharris commented 1 week ago

I'm not sure if this is the correct place to report this but I wanted to make you aware that I've been experiencing connectivity issues to crates.io over IPv6.


; <<>> DiG 9.20.0 <<>> +trace crates.io AAAA @2001:470:20::2
;; global options: +cmd
.                       31769   IN      NS      g.root-servers.net.
.                       31769   IN      NS      l.root-servers.net.
.                       31769   IN      NS      j.root-servers.net.
.                       31769   IN      NS      d.root-servers.net.
.                       31769   IN      NS      e.root-servers.net.
.                       31769   IN      NS      k.root-servers.net.
.                       31769   IN      NS      h.root-servers.net.
.                       31769   IN      NS      a.root-servers.net.
.                       31769   IN      NS      b.root-servers.net.
.                       31769   IN      NS      c.root-servers.net.
.                       31769   IN      NS      i.root-servers.net.
.                       31769   IN      NS      f.root-servers.net.
.                       31769   IN      NS      m.root-servers.net.
.                       31769   IN      RRSIG   NS 8 0 518400 20241104170000 20241022160000 61050 . Hr+GaEPdd15Y1c1E3AWfHy0OiX9cHa8jGzMcEX0Hv+bvXLScBIk1txHK 3gXE2fiqr8CQITYGShd+k9mHHAAJ+2EAdEJR1bscnGZXKPiH/1tuIRjd Mj+0A/IK9gML3iaAETH4o00czYmo0q/T+xvWvDAanqiwos56ojo8wA66 99Fw9HFqknxsC0nMG3ineiE7xMs561TGa8NjEwI6Ttr7aCwv3vLNu5Gt dNIV+xH9W/sXHp/b7S4TI54fcmtzwDLHDaw+yyfOBIY5kyxgoVmn0GJi Qnh6idPrBh66xZgx9IlehnATbBXseg24JsdFT/t/BvBEXh7emAepYw14 nMZWwQ==
;; Received 525 bytes from 2001:470:20::2#53(2001:470:20::2) in 7 ms

io.                     172800  IN      NS      a0.nic.io.
io.                     172800  IN      NS      a2.nic.io.
io.                     172800  IN      NS      b0.nic.io.
io.                     172800  IN      NS      c0.nic.io.
io.                     86400   IN      DS      57355 8 2 95A57C3BAB7849DBCDDF7C72ADA71A88146B141110318CA5BE672057 E865C3E2
io.                     86400   IN      RRSIG   DS 8 1 86400 20241105050000 20241023040000 61050 . OU00tL6oa6lec7P21Rmx58khKSSjBmsPPh4VWOVhQO2eQeUfUUN1Kqsw 9YGcxvvtnRz6iev9A+7c+C9mNviijrPXogCw/wB5c6SzhyL8I8WR3niu Yf2VV0M3q4Df8nVFWZ856haVLovCGlJ3a52FOatQUDn2CvrGUWWIM7lE hCfVMYDxuIIgzYG0j7kvkJ4jvwwrHotaPKGBNyjYeHSGzs+rt81VMGwD CDvF0JfSR3HeS1/z+ctH30nMgFMZTEzuFYyuIEKEE6hVLdbDO4813Zp6 8xLX81tDdCDDsy70Mx+cOGArfgAzOu4aAWtdLzDrPzSFn7/y1k06sDWT yR2d3w==
;; Received 621 bytes from 2001:7fd::1#53(k.root-servers.net) in 7 ms

crates.io.              3600    IN      NS      ns-817.awsdns-38.net.
crates.io.              3600    IN      NS      ns-1543.awsdns-00.co.uk.
crates.io.              3600    IN      NS      ns-217.awsdns-27.com.
crates.io.              3600    IN      NS      ns-1064.awsdns-05.org.
52ko96budrr3nmfkc8cbka2vfqnm5jps.io. 3600 IN NSEC3 1 1 0 73 52KQK0IL9IJ40ET2RNKVBVONSAGJ9T72 NS SOA RRSIG DNSKEY NSEC3PARAM
52ko96budrr3nmfkc8cbka2vfqnm5jps.io. 3600 IN RRSIG NSEC3 8 2 3600 20241113165048 20241023155048 10374 io. TZ9sfAFzIQwAJIygg2oFclq+ROS1yzqJVrgpTf2HWYXVCYaErxoxH5TO TNogtLaYeEeBK/yEGWWQEj12SzKIKsTfwwbuydZ33yvCiDhccYMqn/jV 3ERqdFtkxPOngOoQdYvMvCQkFVHEmfo92CP/mNiMuuxyOn6bXPlY2RRp WcU=
e841shv3mvgstb4nkgtmiq1jf5grjslb.io. 3600 IN NSEC3 1 1 0 73 E842C1DQLJIRG25GRCC9O6OIQLOALM71 NS DS RRSIG
e841shv3mvgstb4nkgtmiq1jf5grjslb.io. 3600 IN RRSIG NSEC3 8 2 3600 20241113002238 20241022232238 10374 io. F6+ggecIT4oQVMXLG1NfOCM7UxhToUBQ9MsrNFKCSv1zv0QUXPWrHZTY 66GBDcJKlTo3jlScii6kJH8NiAOTOAL0ClqIdJaDbsv9snkaDrn6lusd X/ty9j8YIZNuJMLYFF2sHziUhLjwp0/2lD+JYcJpdNWmKX0GuI9JBpL0 FV0=
;; Received 663 bytes from 2a01:8840:9e::17#53(a0.nic.io) in 243 ms

crates.io.              60      IN      AAAA    2600:9000:2046:ca00:c:7ed3:240:93a1
crates.io.              60      IN      AAAA    2600:9000:2046:2e00:c:7ed3:240:93a1
crates.io.              60      IN      AAAA    2600:9000:2046:2800:c:7ed3:240:93a1
crates.io.              60      IN      AAAA    2600:9000:2046:fa00:c:7ed3:240:93a1
crates.io.              60      IN      AAAA    2600:9000:2046:2a00:c:7ed3:240:93a1
crates.io.              60      IN      AAAA    2600:9000:2046:7400:c:7ed3:240:93a1
crates.io.              60      IN      AAAA    2600:9000:2046:b400:c:7ed3:240:93a1
crates.io.              60      IN      AAAA    2600:9000:2046:a200:c:7ed3:240:93a1
crates.io.              60      IN      NS      ns-1064.awsdns-05.org.
crates.io.              60      IN      NS      ns-1543.awsdns-00.co.uk.
crates.io.              60      IN      NS      ns-217.awsdns-27.com.
crates.io.              60      IN      NS      ns-817.awsdns-38.net.
;; Received 402 bytes from 205.251.195.49#53(ns-817.awsdns-38.net) in 23 ms

I'm not able to reach any of these addresses:

$  traceroute6 -I 2600:9000:2046:ca00:c:7ed3:240:93a1
traceroute to 2600:9000:2046:ca00:c:7ed3:240:93a1 (2600:9000:2046:ca00:c:7ed3:240:93a1), 30 hops max, 80 byte packets
 1  2001:470:68c2::1 (2001:470:68c2::1)  0.155 ms  0.138 ms *
 2  tunnel932964.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:fd::1)  10.324 ms * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
$ traceroute6 -i he-ipv6 -I 2600:9000:2046:ca00:c:7ed3:240:93a1
traceroute to 2600:9000:2046:ca00:c:7ed3:240:93a1 (2600:9000:2046:ca00:c:7ed3:240:93a1), 30 hops max, 80 byte packets
 1  tunnel937259.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:2b7::1)  6.097 ms  6.838 ms *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

The above traceroutes are from two different machines in the UK using two different tunnel endpoints (also in the UK, using London endpoints)

I've spoken to the NOC of Hurricane Electric and this seems to be an artifact of some sort of load-balancing Amazon is doing. According to them they have withdrawn the entire :2046: prefix.

It seems crates.io possibly had a DNS typo, or made a change and you seem to have a state DNS record. The third octet is now :20e9:, your dig showed :2046:

Amazon looks to have stopped the :2046: prefix announcement toward us. core2.lon2.he.net ] show ipv6 bgp route 2600:9000:2046::/48 BGP4 : None of the BGP4 routes match the display condition core2.lon2.he.net#

core2.lon2.he.net ] show ipv6 bgp route 2600:9000:20e9::/48 Number of BGP Routes matching display condition : 2 Status A:AGGREGATE B:BEST b:NOT-INSTALLED-BEST C:CONFED_EBGP D:DAMPED E:EBGP H:HISTORY I:IBGP L:LOCAL M:MULTIPATH m:NOT-INSTALLED-MULTIPATH S:SUPPRESSED F:FILTERED s:STALE x:BEST-EXTERNAL Prefix Next Hop MED LocPrf Weight Status 1 2600:9000:20e9::/48 2001:504:0:4:0:1:6509:2 0 100 0 BMI
AS_PATH: 16509 16509

The routing looks good to the :20e9: prefix. If you are still seeing the issue, you can try and clear your DNS cache.

% dig crates.io AAAA

; <<>> DiG 9.10.6 <<>> crates.io AAAA ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54029 ;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;crates.io. IN AAAA

;; ANSWER SECTION: crates.io. 60 IN AAAA 2600:9000:20e9:5800:c:7ed3:240:93a1 crates.io. 60 IN AAAA 2600:9000:20e9:9000:c:7ed3:240:93a1 crates.io. 60 IN AAAA 2600:9000:20e9:1400:c:7ed3:240:93a1 crates.io. 60 IN AAAA 2600:9000:20e9:7c00:c:7ed3:240:93a1 crates.io. 60 IN AAAA 2600:9000:20e9:ec00:c:7ed3:240:93a1 crates.io. 60 IN AAAA 2600:9000:20e9:9a00:c:7ed3:240:93a1 crates.io. 60 IN AAAA 2600:9000:20e9:1e00:c:7ed3:240:93a1 crates.io. 60 IN AAAA 2600:9000:20e9:fe00:c:7ed3:240:93a1

core2.lon2.he.net ] ping ipv6 2600:9000:20e9:5800:c:7ed3:240:93a1 Type Control-c to abort PING 2600:9000:20e9:5800:c:7ed3:240:93a1(2600:9000:20e9:5800:c:7ed3:240:93a1) from 2001:470:0:626::1 56 data bytes 64 bytes from 2600:9000:20e9:5800:c:7ed3:240:93a1: icmp_seq=1 ttl=55 time=84.2 ms 64 bytes from 2600:9000:20e9:5800:c:7ed3:240:93a1: icmp_seq=2 ttl=55 time=84.4 ms 64 bytes from 2600:9000:20e9:5800:c:7ed3:240:93a1: icmp_seq=3 ttl=55 time=84.3 ms 64 bytes from 2600:9000:20e9:5800:c:7ed3:240:93a1: icmp_seq=4 ttl=55 time=85.2 ms 64 bytes from 2600:9000:20e9:5800:c:7ed3:240:93a1: icmp_seq=5 ttl=55 time=84.6 ms

--- 2600:9000:20e9:5800:c:7ed3:240:93a1 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4004ms rtt min/avg/max/mdev = 84.250/84.588/85.204/0.419 ms core2.lon2.he.net#

Is this something that can be looked into? From my end I can ping this :20e9: subnet and others but Amazon DNS still seems to be giving out records for the :2046: prefix I can't reach.

Turbo87 commented 1 week ago

@rust-lang/infra ⬆

jdno commented 6 days ago

crates.io is served through AWS CloudFront, which is where the DNS records point. Just playing with dig a bit myself, I can see that the name resolution rotates through different networks and changes based on geographical location.

We have very little control over the specific DNS records, since we have set up the A and AAAA records for crates.io as aliases for the CloudFront distribution, so everything is handled by AWS internally.

Do you have the problem on other networks as well?

aidanharris commented 6 days ago

I can reach it just fine from other networks. It definitely seems to be some sort of routing issue on Amazon's part.

Right now I am routing 2600:9000:2046::/48 through a VPS over a gre6 tunnel because I got sufficiently annoyed by certain build systems not implementing Happy Eyeballs correctly and taking forever to fallback to IPv4.

aidanharris commented 6 days ago

It's working via France. I don't know how long this will last though. The DNS could change again:

$ traceroute6 -I crates.io
traceroute to crates.io (2600:9000:2171:fe00:c:7ed3:240:93a1), 30 hops max, 80 byte packets
 1  * * *
 2  2001:470:68c2::1 (2001:470:68c2::1)  0.328 ms * *
 3  * * tunnel932964.tunnel.tserv5.lon1.ipv6.he.net (2001:470:1f08:fd::1)  10.049 ms
 4  e0-19.core2.lon2.he.net (2001:470:0:67::1)  11.290 ms  11.525 ms  11.814 ms
 5  * * *
 6  * * *
 7  amazon.par.franceix.net (2001:7f8:54::118)  17.809 ms * *
 8  * * *
 9  * * *
10  * 2a01:578:0:13::8 (2a01:578:0:13::8)  18.017 ms  18.034 ms
11  * * *
12  * * *
13  * * *
14  * * *
15  2600:9000:fff:ff00::401 (2600:9000:fff:ff00::401)  17.356 ms  17.351 ms *
16  * * *
17  * * *
18  2600:9000:2171:fe00:c:7ed3:240:93a1 (2600:9000:2171:fe00:c:7ed3:240:93a1)  17.307 ms  17.303 ms  17.327 ms