ERR_CONNECTION_RESET when connecting with code.jquery.com

bramhaag commented 2 years ago

On Chrome, trying to reach code.jquery.com often results in a ERR_CONNECTION_RESET error. This breaks many pages that use the CDN.

To confirm that it was not a problem with my installation of Chrome, I have tried to following:

Cleared out all site settings, cookies and caches
Reinstalled Chrome
Connected to the domain using Edge

None of these steps allowed me to view code.jquery.com consistently. When refreshing many times, it occasionally happens that the page loads, but this is very rare.

I ran the commands listed here:

$ dig code.jquery.com

; <<>> DiG 9.16.19 <<>> code.jquery.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9017
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;code.jquery.com.               IN      A

;; ANSWER SECTION:
code.jquery.com.        271     IN      CNAME   cds.s5x3j6q5.hwcdn.net.
cds.s5x3j6q5.hwcdn.net. 271     IN      A       69.16.175.42
cds.s5x3j6q5.hwcdn.net. 271     IN      A       69.16.175.10

;; Query time: 6 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sat Aug 07 15:13:32 W. Europe Daylight Time 2021
;; MSG SIZE  rcvd: 112

$ dig AAAA code.jquery.com

; <<>> DiG 9.16.19 <<>> AAAA code.jquery.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56873
;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;code.jquery.com.               IN      AAAA

;; ANSWER SECTION:
code.jquery.com.        265     IN      CNAME   cds.s5x3j6q5.hwcdn.net.
cds.s5x3j6q5.hwcdn.net. 265     IN      AAAA    2001:4de0:ac18::1:a:3a
cds.s5x3j6q5.hwcdn.net. 265     IN      AAAA    2001:4de0:ac18::1:a:1a
cds.s5x3j6q5.hwcdn.net. 265     IN      AAAA    2001:4de0:ac18::1:a:2b
cds.s5x3j6q5.hwcdn.net. 265     IN      AAAA    2001:4de0:ac18::1:a:1b
cds.s5x3j6q5.hwcdn.net. 265     IN      AAAA    2001:4de0:ac18::1:a:2a
cds.s5x3j6q5.hwcdn.net. 265     IN      AAAA    2001:4de0:ac18::1:a:3b

;; Query time: 5 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sat Aug 07 15:14:11 W. Europe Daylight Time 2021
;; MSG SIZE  rcvd: 248

$ curl -vvv -I https://code.jquery.com
* Rebuilt URL to: https://code.jquery.com/
*   Trying 2001:4de0:ac18::1:a:1b...
* TCP_NODELAY set
* Connected to code.jquery.com (2001:4de0:ac18::1:a:1b) port 443 (#0)
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 186 bytes...
* Send failure: Connection was reset
* schannel: failed to send initial handshake data: sent -1 of 186 bytes
* Closing connection 0
* schannel: shutting down SSL/TLS connection with code.jquery.com port 443
* Send failure: Connection was reset
* schannel: failed to send close msg: Failed sending data to the peer (bytes written: -1)
* schannel: clear security context handle
curl: (35) Send failure: Connection was reset

bramhaag commented 2 years ago

Interestingly, re-running the last command (curl -vvv -I https://code.jquery.com) with the -4 flag to force ipv4 does produce some better looking output:


$ curl -vvv -4 -I https://code.jquery.com
* Rebuilt URL to: https://code.jquery.com/
*   Trying 69.16.175.42...
* TCP_NODELAY set
* Connected to code.jquery.com (69.16.175.42) port 443 (#0)
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 186 bytes...
* schannel: sent initial handshake data: sent 186 bytes
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 2/3)
* schannel: encrypted data got 4096
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 4026
* schannel: encrypted data buffer: offset 4026 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 5050 length 5050
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 6074 length 6074
* schannel: encrypted data length: 413
* schannel: encrypted data buffer: offset 413 length 6074
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 2/3)
* schannel: encrypted data got 419
* schannel: encrypted data buffer: offset 832 length 6074
* schannel: sending next handshake data: sending 126 bytes...
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 2/3)
* schannel: encrypted data got 226
* schannel: encrypted data buffer: offset 226 length 6074
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 3/3)
* schannel: stored credential handle in session cache
> HEAD / HTTP/1.1
> Host: code.jquery.com
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 531
* schannel: encrypted data buffer: offset 531 length 103424
* schannel: decrypted data length: 502
* schannel: decrypted data added: 502
* schannel: decrypted data cached: offset 502 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 502 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 502
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Sat, 07 Aug 2021 13:20:22 GMT
Date: Sat, 07 Aug 2021 13:20:22 GMT
< Connection: Keep-Alive
Connection: Keep-Alive
< Cache-Control: max-age=31456841
Cache-Control: max-age=31456841
< Content-Length: 22398
Content-Length: 22398
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Server: nginx
Server: nginx
< X-Powered-By: PHP/5.4.45-1~dotdeb+7.1
X-Powered-By: PHP/5.4.45-1~dotdeb+7.1
< X-Pingback: https://codeorigin.jquery.com/xmlrpc.php
X-Pingback: https://codeorigin.jquery.com/xmlrpc.php
< Link: <//codeorigin.jquery.com/wp-json/>; rel="https://api.w.org/"
Link: <//codeorigin.jquery.com/wp-json/>; rel="https://api.w.org/"
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-HW: 1628342422.dop246.am5.t,1628342422.cds007.am5.shn,1628342422.dop246.am5.t,1628342422.cds227.am5.c
X-HW: 1628342422.dop246.am5.t,1628342422.cds007.am5.shn,1628342422.dop246.am5.t,1628342422.cds227.am5.c

<
* Connection #0 to host code.jquery.com left intact```

bramhaag commented 2 years ago

As a temporary workaround, adding the IPv4 address to the hosts file makes all websites using the CDN work again:

69.16.175.42 code.jquery.com

Krinkle commented 2 years ago

@bramhaag Sorry for the late response. Given the type of issue, this is unlikely to be under our control. However, I am able to consistently reproduce this when using an Amsterdam connection, which is quite worrying indeed.

# Amsterdam (NordVPN)
$ curl -vvv -I https://code.jquery.com --connect-to '::[2001:4de0:ac18::1:a:1a]'
* Connecting to hostname: 2001:4de0:ac18::1:a:1a
*   Trying 2001:4de0:ac18::1:a:1a:443...
* Immediate connect fail for 2001:4de0:ac18::1:a:1a: No route to host
* Closing connection 0
curl: (7) Couldn't connect to server

The same from London, however, works fine:

# London
curl -vvv -I https://code.jquery.com --connect-to '::[2001:4de0:ac18::1:a:1a]'
* Connecting to hostname: 2001:4de0:ac18::1:a:1a
*   Trying 2001:4de0:ac18::1:a:1a:443...
* Connected to 2001:4de0:ac18::1:a:1a (2001:4de0:ac18::1:a:1a) port 443 (#0)
…
HTTP/2 200

I have raised this with Highwinds support.

Krinkle commented 2 years ago

@bramhaag Would you be able to share the IP address from which you experience this issue? This would enable Highwinds support to check their networks for issues in routing from or to you.

To share it privately, send it to infrastructure-team@jquery.com.

bramhaag commented 2 years ago

Now, I do notice that for me, the connection seems to prefer IPv4 by default, thus not affecting my Amsterdam connection by default. That might be due to the VPN's routing choices, or it might be that Highwinds CDN have since mitigated the issue by somehow preferring IPv4. Let me know if it still fails for you by default without the hosts override.

It still fails without the hosts override.

When I forced curl to connect using one of Highwinds's IPv6 addresses, and doing so from an Amsterdam connection, I am unable to get a response, and indeed get the same "No route to host" error. That's quite worrying.

This is a different error than the one I receive. Running the command you've used, I get "Connection was reset":

curl -vvv -I https://code.jquery.com --connect-to '::[2001:4de0:ac18::1:a:1a]'
* Rebuilt URL to: https://code.jquery.com/
*   Trying 2001:4de0:ac18::1:a:1a...
* TCP_NODELAY set
* Connected to code.jquery.com (2001:4de0:ac18::1:a:1a) port 443 (#0)
* schannel: SSL/TLS connection with code.jquery.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 186 bytes...
* Send failure: Connection was reset
* schannel: failed to send initial handshake data: sent -1 of 186 bytes
* Closing connection 0
* schannel: shutting down SSL/TLS connection with code.jquery.com port 443
* Send failure: Connection was reset
* schannel: failed to send close msg: Failed sending data to the peer (bytes written: -1)
* schannel: clear security context handle
curl: (35) Send failure: Connection was reset

@bramhaag Would you be able to share the IP address from which you experience this issue? This would enable Highwinds support to check their networks for issues in routing from or to you.

To share it privately, send it to infrastructure-team@jquery.com.

I have sent an email with my IP address.

Krinkle commented 2 years ago

@bramhaag The folks are Highwinds were not able to find an issue on their side at this time about network paths between them and your IP. They ran a traceroute from a test client provided to them by your Internet provider to 2001:4de0:ac18::1:a:1a and found no issues there, but they'd like yours to compare with, in case it's different. You can email us the result to keep your IP and route private.

Run TCP MTR with the mtr tool on Linux (also available for macOS via Homebrew), like sudo mtr --tcp 2001:4de0:ac18::1:a:1b and have it run for a minute or so and then copy the results. Alternatively without the mtr tool, the result of ping and something like tracert/traceroute/traceroute6 etc would probably help them equally well.

On the other side, perhaps some of the following could help isolate the issue and find just in case something else is happening:

When you run a test like https://ipv6-test.com/, does that report IPv6 as supported? (If not, then the next thing for us to look at would be to figure out why your browser/DNS client are getting an IPv6 address instead of an IPv4 address when resolving code.jquery.com).
If you have access to a mobile connection, can you access code.jquery.com there from its default browser whilst on the same WiFi as your computer? And what about the inverse, if you use your phone as hotspot for your computer, are you able to access the site then using either Chrome or curl?

bramhaag commented 2 years ago

As of tonight, for whatever reason IPv4 is being preferred for me when connecting to code.jquery.com. This change happened in the middle of my testing without changing anything on my side. I confirmed this with the IPvFoo extension for Chrome. Connecting using IPv6 is still broken though.

Run TCP MTR with the mtr tool on Linux (also available for macOS via Homebrew), like sudo mtr --tcp 2001:4de0:ac18::1:a:1b and have it run for a minute or so and then copy the results. Alternatively without the mtr tool, the result of ping and something like tracert/traceroute/traceroute6 etc would probably help them equally well.

I have emailed the output of tracert and ping.

When you run a test like https://ipv6-test.com/, does that report IPv6 as supported?

It does:

IPv6: Supported
Address: redacted
Type: Native IPv6
SLAAC: No
ICMP: Not tested

If you have access to a mobile connection, can you access code.jquery.com there from its default browser whilst on the same WiFi as your computer?

This works, but possibly because IPv4 is now preferred for me.

And what about the inverse, if you use your phone as hotspot for your computer, are you able to access the site then using either Chrome or curl?

Still unable to access the site with curl

Krinkle commented 2 years ago

@bramhaag Thanks for sticking with me. They've checked the route and so far have not uncovered any issue. They believe an MTR report would help more there. Would you be able to capture and send one of those as well? (If you're on Windows, WinMTR might work, which was recommended to me by Highwinds support.)

bramhaag commented 2 years ago

I have emailed the output of WinMTR. The original does not support IPv6, so I've used a fork that does.

Krinkle commented 2 years ago

@bramhaag Thanks. Unfortunately still no luck pinpointing the issue. There is growing suspicion that it is an issue with either the device or something from your ISP that is somehow very localised to a small subset of customers from that ISP.

There's two additional things Highwinds support would like you to try:

curl with HTTP (instead of HTTPS, so no TLS).

curl -vvv -I http://code.jquery.com

Trying each of their IPv6 addresses explicitly, and sharing any output where it differed from what we've seen above with HTTP and previously with HTTPS.

curl -vvv -I http://code.jquery.com --connect-to '::[2001:4de0:ac18::1:a:1a]'
curl -vvv -I https://code.jquery.com --connect-to '::[2001:4de0:ac18::1:a:1a]'

For each of:

2001:4de0:ac18::1:a:1a
2001:4de0:ac18::1:a:1b
2001:4de0:ac18::1:a:2a
2001:4de0:ac18::1:a:3b
2001:4de0:ac18::1:a:2b
2001:4de0:ac18::1:a:3

bramhaag commented 2 years ago

Sorry for the delay. In the past few days, something must have changed as I can now reach code.jquery.com from all my devices on the affected network without any changes on my end. Querying the IP addresses listed above with curl now gives me the desired output.

Out of curiosity, are you still able to reproduce the issue using your Amsterdam VPN @Krinkle?

Krinkle commented 2 years ago

@bramhaag I am, but I have since realized that that was due to the VPN in question not supporting IPv6 (it didn't pass the test either), and it was only reproducible there when explicitly forcing the connection over an IPv6 address.

I'll close this for now. Thanks again for sticking with us, and don't hesitate to reach out again. 🙂

ae2x5yl commented 2 years ago

I also got that problem with connetion resets while loading javascript from code.jquery.com

GET https://code.jquery.com/jquery-3.4.1.slim.min.js net::ERR_CONNECTION_RESET

When I repeat reloading the website then it works after 10-20 reloads.

I added the ip4 address to the hosts file, then it works.

Looks like a DNS issue for code.jquery.com. It is solved to

ip6 address cds.s5x3j6q5.hwcdn.net [2001:4de0:ac18::1:a:3b] ip4 address 69.16.175.10 code.jquery.com

The ip6 address is preferred while solving with ping -a. (boths pings are always working) (for jquery.com ip4 is preferred!?)

motyc commented 2 years ago

I must add myself to the list of people experiencing this specific issue. Only workaround that helped so far was editing the hosts file as suggested above... It is happening on the single device (other in the same network work just fine).

MackSix commented 2 years ago

For me, this is caused by the Intel software, "Killer Network Services."

See here: https://github.com/jquery/codeorigin.jquery.com/issues/80#issuecomment-1059703212

jquery / codeorigin.jquery.com

ERR_CONNECTION_RESET when connecting with code.jquery.com #77