search

lxc / linuxcontainers.org

The linuxcontainers.org website
https://linuxcontainers.org
Other
96 stars 118 forks source link

images website is hanging #465

Closed jamshid closed 4 years ago

jamshid commented 4 years ago

Sorry not sure where to report this but your website is hanging, breaking lxc image list images: on chromeos. I think your site is trying to redirect to uk.images.linuxcontainers.org but sometimes it hangs, e.g.

% curl -v https://images.linuxcontainers.org/streams/v1/index.json
*   Trying 2001:67c:1560:8001::21...
* TCP_NODELAY set
* Connected to images.linuxcontainers.org (2001:67c:1560:8001::21) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=GB; L=London; O=Canonical Group Ltd; OU=IS; CN=images.linuxcontainers.org
*  start date: Aug 27 00:00:00 2020 GMT
*  expire date: Sep  1 12:00:00 2021 GMT
*  subjectAltName: host "images.linuxcontainers.org" matched cert's "images.linuxcontainers.org"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> GET /streams/v1/index.json HTTP/1.1
> Host: images.linuxcontainers.org
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Date: Tue, 24 Nov 2020 18:30:50 GMT
< Server: Apache/2.4.7 (Ubuntu)
< Strict-Transport-Security: max-age=31536000
< Location: https://uk.images.linuxcontainers.org/streams/v1/index.json
< Content-Length: 359
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://uk.images.linuxcontainers.org/streams/v1/index.json">here</a>.</p>
<hr>
<address>Apache/2.4.7 (Ubuntu) Server at images.linuxcontainers.org Port 443</address>
</body></html>
* Connection #0 to host images.linuxcontainers.org left intact
* Closing connection 0

% curl -v https://images.linuxcontainers.org/streams/v1/index.json
*   Trying 2001:67c:1562::41...
* TCP_NODELAY set
* Connected to images.linuxcontainers.org (2001:67c:1562::41) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
stgraber commented 4 years ago

Forcing a test against 2001:67c:1562::41 works fine here.

*   Trying 2001:67c:1562::41:443...
* TCP_NODELAY set
* Connected to images.linuxcontainers.org (2001:67c:1562::41) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=GB; L=London; O=Canonical Group Ltd; OU=IS; CN=images.linuxcontainers.org
*  start date: Aug 27 00:00:00 2020 GMT
*  expire date: Sep  1 12:00:00 2021 GMT
*  subjectAltName: host "images.linuxcontainers.org" matched cert's "images.linuxcontainers.org"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*  SSL certificate verify ok.
> GET /streams/v1/index.json HTTP/1.1
> Host: images.linuxcontainers.org
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Date: Tue, 24 Nov 2020 18:43:31 GMT
< Server: Apache/2.4.7 (Ubuntu)
< Strict-Transport-Security: max-age=31536000
< Location: https://us.images.linuxcontainers.org/streams/v1/index.json
< Content-Length: 359
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://us.images.linuxcontainers.org/streams/v1/index.json">here</a>.</p>
<hr>
<address>Apache/2.4.7 (Ubuntu) Server at images.linuxcontainers.org Port 443</address>
</body></html>
* Connection #0 to host images.linuxcontainers.org left intact

We closely monitor both of those servers over both IPv4 and IPv6, so it's not a server-side hang but instead some kind of network issue between you and that server.

I'll send a link to this issue to the network admins just in case there's something on our side that needs to be looked at. Can you do mtr --report-wide 2001:67c:1562::41 (requires mtr or mtr-tiny be installed) and post the result here?

stgraber commented 4 years ago

(Closing as not an issue with the website or anything we directly control)

jamshid commented 4 years ago

Ok thanks, yeah it does seem related to my home network (ATT Uverse).

If anything looks unusual or you have any idea what could be wrong let me know.

From penguin terminal on chromeos:

Start: 2020-11-24T13:15:08-0600
HOST: penguin                                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2600:1700:291:e470:e4fb:fcff:fe70:a92c  0.0%    10    0.2   0.2   0.2   0.3   0.0
  2.|-- 2600:1700:291:e470:d82b:ad87:1069:c708  0.0%    10    0.5   0.9   0.5   2.0   0.4
  3.|-- ???                                    100.0    10    0.0   0.0   0.0   0.0   0.0
  4.|-- 2001:506:6000:100:76:253:217:29         0.0%    10    3.1   3.3   2.9   4.1   0.3
  5.|-- 2001:506:6000:100:76:253:217:15         0.0%    10    3.0   3.7   3.0   5.3   0.8
  6.|-- 2001:1890:ff:ffff:12:123:18:233        60.0%    10   13.5  13.6  11.0  15.4   1.9
  7.|-- ae-10.edge1.Dallas2.Level3.net          0.0%    10   13.9  13.0   8.2  25.0   6.2
  8.|-- lo-0-v6.ear2.London1.Level3.net         0.0%    10  110.3 110.5 110.0 111.5   0.5
  9.|-- SOURCE-MANA.edge6.London1.Level3.net    0.0%    10  110.4 118.4 110.3 188.8  24.7
 10.|-- 2001:67c:1360:1::94                    40.0%    10  187.3 197.9 185.8 233.8  18.6
 11.|-- gible.canonical.com                     0.0%    10  187.6 215.3 185.6 277.0  36.9

A traceroute from my ATT router:

traceroute to 2001:67c:1562::41 (2001:67c:1562::41) from 2600:1700:291:e470::1, 30 hops max, 64 byte packets
 1  2001:506:6000:100:76:253:217:29 (2001:506:6000:100:76:253:217:29)  1.245 ms  1.383 ms  0.925 ms
 2  2001:506:6000:100:76:253:217:15 (2001:506:6000:100:76:253:217:15)  1.241 ms  1.185 ms  1.259 ms
 3  *  *  *
 4  ae-10.edge1.Dallas2.Level3.net (2001:1900:4:3::6a9)  6.869 ms  6.699 ms  6.557 ms
 5  *  *  *
 6  SOURCE-MANA.edge6.London1.Level3.net (2001:1900:5:2:2::44b2)  107.988 ms  108.131 ms  108.017 ms
 7  *  2001:67c:1360:1::94 (2001:67c:1360:1::94)  198.661 ms  *
 8  gible.canonical.com (2001:67c:1562::41)  180.961 ms  182.838 ms  181.971 ms
stgraber commented 4 years ago

Ok, can you do tracepath -6 gible.canonical.com too? I'm wondering if there's an MTU issue somewhere along the way.

jamshid commented 4 years ago

Thanks! MTU might be misconfigured on my chromebook or router?

penguin:~$ tracepath -6 gible.canonical.com
 1?: [LOCALHOST]                        0.014ms pmtu 1500
 1:  2600:1700:291:e470:e4fb:fcff:fe70:a92c                0.048ms 
 1:  2600:1700:291:e470:e4fb:fcff:fe70:a92c                0.034ms 
 2:  2600:1700:291:e470:d82b:ad87:1069:c708                0.375ms 
 3:  no reply
 4:  2001:506:6000:100:76:253:217:29                       5.863ms 
 5:  2001:506:6000:100:76:253:217:15                       3.427ms 
 6:  2001:1890:ff:ffff:12:123:18:233                      11.221ms asymm  7 
 7:  ae-10.edge1.Dallas2.Level3.net                       10.252ms asymm  9 
 8:  no reply
 9:  SOURCE-MANA.edge6.London1.Level3.net                203.961ms asymm 13 
10:  SOURCE-MANA.edge6.London1.Level3.net                204.289ms pmtu 1350
10:  no reply
11:  no reply
12:  no reply
13:  no reply
14:  no reply
15:  no reply
16:  no reply
17:  no reply
18:  no reply
19:  no reply
20:  no reply
21:  no reply
22:  no reply
23:  no reply
24:  no reply
25:  no reply
26:  no reply
27:  no reply
28:  no reply
29:  no reply
30:  no reply
     Too many hops: pmtu 1350
     Resume: pmtu 1350
jamshid commented 4 years ago

FYI turning IPv6 "Off" on my ATT UVerse ARRIS BGW210-700 2.8.7 seems to WORKAROUND the problem. The configuration page previously showed Router Advertisement MTU 1500. Now that it's off it shows 1472. The comment says this, I wonder if the old default was 1500 and that was causing my problems.

Router Advertisement MTU: This MTU indicates the maximum packet size before the device will split an IPv6 packet into multiple packets. If IPv6 6rd is configured, this MTU is normally at least 20 bytes less than the base MTU. In most cases, this parameter is not configurable by the user.

stgraber commented 4 years ago

Hmm, yeah, if your provider is using 6rd and your modem was incorrectly enforcing a 1500 MTU, that could explain it.