search

pypi / support

Issue tracker for support requests related to using https://pypi.org
95 stars 48 forks source link

Requests via squid proxies are being blocked from some regions of Google Cloud #405

Closed markroth8 closed 4 years ago

markroth8 commented 4 years ago

My Platform

We are experiencing timeouts when trying to https_proxy=http://squid-proxy-host:3128/ curl https://pypi.org/ from a client going through a squid proxy at squid-proxy-host. This only affected proxies in some of our Google Cloud regions until yesterday but now all of our proxied requests are timing out.

$ curl --version
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1d zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL 

squid: 4.10

Our squid is presently configured as a MITM proxy but the same behavior seems to happen with our without the MITM.

Network telemetry

Setup:

Client (in Google Cloud) --> Squid proxy (in Google Cloud) --> https://pypi.org/

What is interesting is that a direct curl https://pypi.org/ coming from the squid host works, but the client request fails with a timeout. Hitting other websites works fine - it is only pypi.org that is failing for us.

DNS Resolution

(From the proxy host)

dig pypi.org A

; <<>> DiG 9.11.3-1ubuntu1.12-Ubuntu <<>> pypi.org A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28995
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;pypi.org.          IN  A

;; ANSWER SECTION:
pypi.org.       21599   IN  A   151.101.0.223
pypi.org.       21599   IN  A   151.101.192.223
pypi.org.       21599   IN  A   151.101.64.223
pypi.org.       21599   IN  A   151.101.128.223

;; Query time: 2 msec
;; SERVER: 169.254.169.254#53(169.254.169.254)
;; WHEN: Fri May 22 22:28:49 UTC 2020
;; MSG SIZE  rcvd: 101
dig pypi.org AAAA
<Replace with your output>

Traceroutes

IPv4

traceroute pypi.org
traceroute to pypi.org (151.101.0.223), 30 hops max, 60 byte packets
 1  172.17.0.1 (172.17.0.1)  0.022 ms  0.006 ms  0.006 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

HTTPS Requests

IPv4

(from the client)

Some proxies are not blocked and work fine. The same exact proxy deployed in a different Google Cloud region causes a timeout.

$ https_proxy=<bad-proxy> curl https://pypi.org/pypi/pip/json
<times out>

$ https_proxy=<good-proxy> curl https://pypi.org/pypi/pip/json
<works>

(from both good and bad proxy servers, the connection works:)

curl -vvv -I --ipv4 https://pypi.org/pypi/pip/json

# curl -vvv -I --ipv4 https://pypi.org/pypi/pip/json
*   Trying 151.101.128.223...
* TCP_NODELAY set
* Connected to pypi.org (151.101.128.223) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=Delaware; serialNumber=3359300; C=US; ST=New Hampshire; L=Wolfeboro; O=Python Software Foundation; CN=www.python.org
*  start date: Sep 18 00:00:00 2018 GMT
*  expire date: Oct 14 12:00:00 2020 GMT
*  subjectAltName: host "pypi.org" matched cert's "pypi.org"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x563eeca6c580)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> HEAD /pypi/pip/json HTTP/2
> Host: pypi.org
> User-Agent: curl/7.58.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 200 
HTTP/2 200 
< access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
< access-control-allow-methods: GET
access-control-allow-methods: GET
< access-control-allow-origin: *
access-control-allow-origin: *
< access-control-expose-headers: X-PyPI-Last-Serial
access-control-expose-headers: X-PyPI-Last-Serial
< access-control-max-age: 86400
access-control-max-age: 86400
< cache-control: max-age=900, public
cache-control: max-age=900, public
< content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ *.fastly-insights.com sentry.io https://api.pwnedpasswords.com https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.ingress.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com https://cdn.ravenjs.com; style-src 'self' fonts.googleapis.com; worker-src *.fastly-insights.com
content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ *.fastly-insights.com sentry.io https://api.pwnedpasswords.com https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.ingress.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com https://cdn.ravenjs.com; style-src 'self' fonts.googleapis.com; worker-src *.fastly-insights.com
< content-type: application/json
content-type: application/json
< etag: "1bScw84sINFW/pIshXUiKQ"
etag: "1bScw84sINFW/pIshXUiKQ"
< referrer-policy: origin-when-cross-origin
referrer-policy: origin-when-cross-origin
< server: nginx/1.13.9
server: nginx/1.13.9
< x-pypi-last-serial: 7292768
x-pypi-last-serial: 7292768
< accept-ranges: bytes
accept-ranges: bytes
< date: Fri, 22 May 2020 22:31:57 GMT
date: Fri, 22 May 2020 22:31:57 GMT
< x-served-by: cache-bwi5125-BWI, cache-wdc5569-WDC
x-served-by: cache-bwi5125-BWI, cache-wdc5569-WDC
< x-cache: HIT, HIT
x-cache: HIT, HIT
< x-cache-hits: 1, 1
x-cache-hits: 1, 1
< x-timer: S1590186717.113120,VS0,VE1
x-timer: S1590186717.113120,VS0,VE1
< vary: Accept-Encoding, Accept-Encoding
vary: Accept-Encoding, Accept-Encoding
< strict-transport-security: max-age=31536000; includeSubDomains; preload
strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-frame-options: deny
x-frame-options: deny
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-permitted-cross-domain-policies: none
x-permitted-cross-domain-policies: none
< content-length: 99851
content-length: 99851

< 
* Connection #0 to host pypi.org left intact

TLS Debug

IPv4

(from proxy server)

echo -n | openssl s_client -4 -connect pypi.org:443
# echo -n | openssl s_client -4 -connect pypi.org:443
CONNECTED(00000005)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA
verify return:1
depth=0 businessCategory = Private Organization, jurisdictionC = US, jurisdictionST = Delaware, serialNumber = 3359300, C = US, ST = New Hampshire, L = Wolfeboro, O = Python Software Foundation, CN = www.python.org
verify return:1
---
Certificate chain
 0 s:businessCategory = Private Organization, jurisdictionC = US, jurisdictionST = Delaware, serialNumber = 3359300, C = US, ST = New Hampshire, L = Wolfeboro, O = Python Software Foundation, CN = www.python.org
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA
 1 s:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIIczCCB1ugAwIBAgIQDl7PGBeDAG2brEU2EfVJEjANBgkqhkiG9w0BAQsFADB1
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMTQwMgYDVQQDEytEaWdpQ2VydCBTSEEyIEV4dGVuZGVk
IFZhbGlkYXRpb24gU2VydmVyIENBMB4XDTE4MDkxODAwMDAwMFoXDTIwMTAxNDEy
MDAwMFowgdgxHTAbBgNVBA8MFFByaXZhdGUgT3JnYW5pemF0aW9uMRMwEQYLKwYB
BAGCNzwCAQMTAlVTMRkwFwYLKwYBBAGCNzwCAQITCERlbGF3YXJlMRAwDgYDVQQF
EwczMzU5MzAwMQswCQYDVQQGEwJVUzEWMBQGA1UECBMNTmV3IEhhbXBzaGlyZTES
MBAGA1UEBxMJV29sZmVib3JvMSMwIQYDVQQKExpQeXRob24gU29mdHdhcmUgRm91
bmRhdGlvbjEXMBUGA1UEAxMOd3d3LnB5dGhvbi5vcmcwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQD2roGTDc+m3g3mReIf7j5rzLovnk3Zggvy2kWWMnDB
Yz2m6sgxOL1pkzIu7YNe4noH7ZRym0OIQjZbDIleB7SOGAoD2BBv+Mv79HtedJ3d
vhXwmgRenlNxBxhDcaVLDONVC9Ir3Ft46uuGIrXnKB8L1KZV6h8IFC2G3GZXJtkw
EojXdmJFzuRrKMRi0cLPlF60upRISIj3jg9kK4D+f0xYrsQAJvK0veqRsNQ8bXPE
YR8kG4Paj7rOeh3FVGxrKOKNpoI+kV6EqOVIhJY698P7EbDLGjG2Im9P6VFmwXod
YAZ2CTFaMlrC0FljAO/FKeQKR29vitthLMutcd+AkoDDAgMBAAGjggSZMIIElTAf
BgNVHSMEGDAWgBQ901Cl1qCt7vNKYApl0yHU+PjWDzAdBgNVHQ4EFgQUUTsyHAXZ
y6d2HWn+8MZNUhBCkEwwggFCBgNVHREEggE5MIIBNYIOd3d3LnB5dGhvbi5vcmeC
D2RvY3MucHl0aG9uLm9yZ4IPYnVncy5weXRob24ub3Jngg93aWtpLnB5dGhvbi5v
cmeCDWhnLnB5dGhvbi5vcmeCD21haWwucHl0aG9uLm9yZ4IPcHlwaS5weXRob24u
b3JnghRwYWNrYWdpbmcucHl0aG9uLm9yZ4IQbG9naW4ucHl0aG9uLm9yZ4ISZGlz
Y3Vzcy5weXRob24ub3Jnggx1cy5weWNvbi5vcmeCB3B5cGkuaW+CDGRvY3MucHlw
aS5pb4IIcHlwaS5vcmeCDWRvY3MucHlwaS5vcmeCD2RvbmF0ZS5weXBpLm9yZ4IT
ZGV2Z3VpZGUucHl0aG9uLm9yZ4ITd3d3LmJ1Z3MucHl0aG9uLm9yZ4IKcHl0aG9u
Lm9yZzAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUF
BwMCMHUGA1UdHwRuMGwwNKAyoDCGLmh0dHA6Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9z
aGEyLWV2LXNlcnZlci1nMi5jcmwwNKAyoDCGLmh0dHA6Ly9jcmw0LmRpZ2ljZXJ0
LmNvbS9zaGEyLWV2LXNlcnZlci1nMi5jcmwwSwYDVR0gBEQwQjA3BglghkgBhv1s
AgEwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cuZGlnaWNlcnQuY29tL0NQUzAH
BgVngQwBATCBiAYIKwYBBQUHAQEEfDB6MCQGCCsGAQUFBzABhhhodHRwOi8vb2Nz
cC5kaWdpY2VydC5jb20wUgYIKwYBBQUHMAKGRmh0dHA6Ly9jYWNlcnRzLmRpZ2lj
ZXJ0LmNvbS9EaWdpQ2VydFNIQTJFeHRlbmRlZFZhbGlkYXRpb25TZXJ2ZXJDQS5j
cnQwDAYDVR0TAQH/BAIwADCCAX8GCisGAQQB1nkCBAIEggFvBIIBawFpAHcA7ku9
t3XOYLrhQmkfq+GeZqMPfl+wctiDAMR7iXqo/csAAAFl7l3MWwAABAMASDBGAiEA
qfAJSfOHG4r8YvzTkZsr8cEXFcnFIns40+JXVdgY0vsCIQCvnB+YExtMRQVQXONc
glOTsIYmNgYPrtyHYsTj/Xua5QB3AFYUBpov18Ls0/XhvUSyPsdGdrm8mRFcwO+U
mFXWidDdAAABZe5dzHoAAAQDAEgwRgIhAIF6xXakKVdREciK8aM2z1c71eiU8qF/
UCZbl4sEfLTQAiEA870pR9Hazod3FgZszr9itk8sYPLoQjV9/WENl0HXXGwAdQC7
2d+8H4pxtZOUI5eqkntHOFeVCqtS6BqQlmQ2jh7RhQAAAWXuXcwzAAAEAwBGMEQC
IAeIvcaqPATJrCo3+ceBlVbsyPJcDPM7QGLpaRPFBduQAiBHNfcsBLaelS9bUmfU
R0VjJLoTA39AJAj8oU6iAzruHDANBgkqhkiG9w0BAQsFAAOCAQEAwH5+PXougfo5
qMVIE65Dei/CEb4ahZfoMvvlJRvuNAI/ARWYQSHh6IKXdxFxIE5hCC/NNtdAcc1p
CoZ2+IM/x0cGBjHZegSjhXfQy+w7twfgyeTSNalV2jzKQ0Yv/JvfI9qiMdhEQbfL
qaQ6Nj/js8uvQqWf6w8yo7hAzKs1jTF7Wy/cM0lvqNocDWYROhAVcI+jSMKwlcLv
75xAbOZqw2D483mkQizVj7wQ1fYP3Tr0wfvFNeoIAPUXLrEHiEmHWA0h+U9KEgVf
VQ3PvzItecKWFI+0d9RpNupc1HdipBCySvnNa1LcG2AviYXpIsbo8EBvbj0HcEVx
7NEANHNX2g==
-----END CERTIFICATE-----
subject=businessCategory = Private Organization, jurisdictionC = US, jurisdictionST = Delaware, serialNumber = 3359300, C = US, ST = New Hampshire, L = Wolfeboro, O = Python Software Foundation, CN = www.python.org

issuer=C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA

---
No client certificate CA names sent
Peer signing digest: SHA512
Peer signature type: RSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4025 bytes and written 403 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 901EF04A764DB327FC92FF4052DC2263D4885A72AFFA1A3A2F70C8727BE27159
    Session-ID-ctx: 
    Master-Key: 095D84DBFFEB177D9F2C3F242C18E23673D0B2EFCDEC86FFE4901486CA3B1591779ABAF4212ABE8C7E7701366995041F
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 3d 80 0e 15 4b ba 35 69-20 cc 62 05 bb 58 44 82   =...K.5i .b..XD.
    0010 - 87 3c 75 86 43 ff 0d b2-53 39 ab 38 4a 64 a5 ae   .<u.C...S9.8Jd..
    0020 - 89 c3 e9 24 cf 79 85 c4-05 92 8c 3b ee a0 47 d8   ...$.y.....;..G.
    0030 - 92 b4 1e a3 59 1f c3 1b-38 95 8d 20 8b 5c 41 8a   ....Y...8.. .\A.
    0040 - af d2 9c 79 38 95 bd cd-88 eb e1 08 d6 87 11 a7   ...y8...........
    0050 - cd 5c ed 7c f3 53 84 2b-e4 14 f3 fd 34 2b 7d ef   .\.|.S.+....4+}.
    0060 - 05 f5 f2 4c 1b 67 6d 21-d4 07 d3 49 da 8b d6 d0   ...L.gm!...I....
    0070 - ec e2 10 9b 7c 4f 85 2a-4b 51 69 b6 89 f3 f7 a0   ....|O.*KQi.....
    0080 - dd e7 71 d8 8a 66 60 13-9f f9 f1 d2 d8 d3 af fb   ..q..f`.........
    0090 - a9 ea 3c 90 43 bf 85 46-3b 11 85 e4 c5 c3 4f 58   ..<.C..F;.....OX
    00a0 - 48 8c 34 e5 aa 65 50 17-95 95 65 f0 f0 2b 02 58   H.4..eP...e..+.X

    Start Time: 1590187138
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
DONE
markroth8 commented 4 years ago

Duplicate - github was returning a 500 error and when I resubmitted this issue appeared 3 times.