search

pypi / support

Issue tracker for support requests related to using https://pypi.org
95 stars 48 forks source link

Test PyPI: 104 connection reset by peer in GitHub Actions runner #2317

Closed zanieb closed 1 year ago

zanieb commented 1 year ago

My Platform

Hi! Upload of my package is failing in a GitHub Actions runner with the error: 104 Connection reset by peer. I'm also seeing

WARNING Received "503: Backend is unhealthy

We are using the pypa/gh-action-pypi-publish@release/v1 action which runs on Python 3.9 and uses urllib3.

Example run: https://github.com/PrefectHQ/prefect/actions/runs/3292121693/jobs/5427224009

This persisted after multiple retry attempts. Each run attempted 5 uploads.

I wrote a GitHub action to collect debug information as requested below. See the output at for all three hosts https://github.com/PrefectHQ/prefect/actions/runs/3292376903 (I copied the testpypi output below).

Fastly Debug

DNS Resolution

$ dig test.pypi.org A

; <<>> DiG 9.16.1-Ubuntu <<>> test.pypi.org A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15208
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;test.pypi.org.         IN  A

;; ANSWER SECTION:
test.pypi.org.      281 IN  CNAME   dualstack.r.ssl.global.fastly.net.
dualstack.r.ssl.global.fastly.net. 29 IN A  146.75.29.63

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Oct 20 19:30:28 UTC 2022
;; MSG SIZE  rcvd: 105

$ dig test.pypi.org AAAA

; <<>> DiG 9.16.1-Ubuntu <<>> test.pypi.org AAAA
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33517
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;test.pypi.org.         IN  AAAA

;; ANSWER SECTION:
test.pypi.org.      280 IN  CNAME   dualstack.r.ssl.global.fastly.net.
dualstack.r.ssl.global.fastly.net. 7 IN AAAA    2a04:4e42:77::319

;; Query time: 4 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Oct 20 19:30:28 UTC 2022
;; MSG SIZE  rcvd: 117

Traceroutes / IPv4

$ traceroute test.pypi.org
traceroute to test.pypi.org (146.75.29.63), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Traceroutes / IPv6 (If available)

$ traceroute6 pypi.org

$ traceroute6 files.pythonhosted.org

HTTPS Requests / IPv4

$ curl -vvv -I --ipv4 https://test.pypi.org/pypi/pip/json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 146.75.29.63:443...
* TCP_NODELAY set
* Connected to test.pypi.org (146.75.29.63) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [106 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2851 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=pypi.org
*  start date: Jul 26 19:45:14 2022 GMT
*  expire date: Aug 27 19:45:13 2023 GMT
*  subjectAltName: host "test.pypi.org" matched cert's "*.pypi.org"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Atlas R3 DV TLS CA 2022 Q3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55af5038b8c0)
} [5 bytes data]
> HEAD /pypi/pip/json HTTP/2
> Host: test.pypi.org
> user-agent: curl/7.68.0
> accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 503 
< server: Varnish
< retry-after: 0
< content-type: text/html; charset=utf-8
< accept-ranges: bytes
< date: Thu, 20 Oct 2022 19:30:58 GMT
< x-served-by: cache-iad-kiad7000114-IAD
< x-cache: MISS
< x-cache-hits: 0
< x-timer: S1666294259.537398,VS0,VE79
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-frame-options: deny
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
< x-permitted-cross-domain-policies: none
< content-length: 22461
< 

  0 22461    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/2 503 
server: Varnish
retry-after: 0
content-type: text/html; charset=utf-8
accept-ranges: bytes
date: Thu, 20 Oct 2022 19:30:58 GMT
x-served-by: cache-iad-kiad7000114-IAD
x-cache: MISS
x-cache-hits: 0
x-timer: S1666294259.537398,VS0,VE79
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: deny
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-permitted-cross-domain-policies: none
content-length: 22461

* Connection #0 to host test.pypi.org left intact

HTTPS Requests / IPv6 (If available)

$ curl -vvv -I --ipv6 https://pypi.org/pypi/pip/json

$ curl -vvv -I --ipv6 https://files.pythonhosted.org/packages/ae/e8/2340d46ecadb1692a1e455f13f75e596d4eab3d11a57446f08259dee8f02/pip-10.0.1.tar.gz

TLS Debug / IPv4

$ echo -n | openssl s_client -4 -connect test.pypi.org:443
depth=2 OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = BE, O = GlobalSign nv-sa, CN = GlobalSign Atlas R3 DV TLS CA 2022 Q3
verify return:1
depth=0 CN = pypi.org
verify return:1
CONNECTED(00000003)
DONE
---
Certificate chain
 0 s:CN = pypi.org
   i:C = BE, O = GlobalSign nv-sa, CN = GlobalSign Atlas R3 DV TLS CA 2022 Q3
 1 s:C = BE, O = GlobalSign nv-sa, CN = GlobalSign Atlas R3 DV TLS CA 2022 Q3
   i:OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGfzCCBWegAwIBAgIQAe6p6b6zdP9u5myWWPuk2zANBgkqhkiG9w0BAQsFADBY
MQswCQYDVQQGEwJCRTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTEuMCwGA1UE
AxMlR2xvYmFsU2lnbiBBdGxhcyBSMyBEViBUTFMgQ0EgMjAyMiBRMzAeFw0yMjA3
MjYxOTQ1MTRaFw0yMzA4MjcxOTQ1MTNaMBMxETAPBgNVBAMMCHB5cGkub3JnMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA8o44EkMKZTdtsEtjJs1caeZE
n3YE0fNgO0H7PxbBoNsWFYtupoRz1FRdnQLThGOVg53Qo/nhBRQAxvsX4aOxk15N
PIC8Irz+XYahv4cL9lTa+oQkQINgyG7G4ohXKZDRpTnRZMOCh+VyYANq0Ou5G855
tZv1Q2iFtvxXzr2VGLZTSbqZCemaO0zBaVxSTOLrMbRDUmRh3nnCQIq61Agg+6oe
tT7OBmGQNyG998iA7/R4DcNrFFw1zM64c24v6xM+3Lq3yx6x6OCbFuxB6waJuBUh
eP8OUIlaPl6IYzKWPHZ8p5iKIfTYo4iGKAwdJF241sivWVXngxezS3M+kGSUdQID
AQABo4IDiDCCA4QwPgYDVR0RBDcwNYIIcHlwaS5vcmeCCioucHlwaS5vcmeCDHd3
dy5weXBpLm9yZ4IPZG9uYXRlLnB5cGkub3JnMA4GA1UdDwEB/wQEAwIFoDAdBgNV
HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwHQYDVR0OBBYEFEfow+xITtwxOc96
M4M5FTpT5iC1MFcGA1UdIARQME4wCAYGZ4EMAQIBMEIGCisGAQQBoDIKAQMwNDAy
BggrBgEFBQcCARYmaHR0cHM6Ly93d3cuZ2xvYmFsc2lnbi5jb20vcmVwb3NpdG9y
eS8wDAYDVR0TAQH/BAIwADCBngYIKwYBBQUHAQEEgZEwgY4wQAYIKwYBBQUHMAGG
NGh0dHA6Ly9vY3NwLmdsb2JhbHNpZ24uY29tL2NhL2dzYXRsYXNyM2R2dGxzY2Ey
MDIycTMwSgYIKwYBBQUHMAKGPmh0dHA6Ly9zZWN1cmUuZ2xvYmFsc2lnbi5jb20v
Y2FjZXJ0L2dzYXRsYXNyM2R2dGxzY2EyMDIycTMuY3J0MB8GA1UdIwQYMBaAFPqR
OWOa+60QJOW+tbnaq9nERmmrMEgGA1UdHwRBMD8wPaA7oDmGN2h0dHA6Ly9jcmwu
Z2xvYmFsc2lnbi5jb20vY2EvZ3NhdGxhc3IzZHZ0bHNjYTIwMjJxMy5jcmwwggF/
BgorBgEEAdZ5AgQCBIIBbwSCAWsBaQB2AK33vvp8/xDIi509nB4+GGq0Zyldz7EM
JMqFhjTr3IKKAAABgjwKxXoAAAQDAEcwRQIgS1S5S5j9RshJIiN7ohMvHQk+fRp/
TLACc1FuxFzukoMCIQC0kSLRxGVusXukcMXlu3VDzTyUvndArct7LrNOWX7JlAB2
ALNzdwfhhFD4Y4bWBancEQlKeS2xZwwLh9zwAw55NqWaAAABgjwKxfwAAAQDAEcw
RQIhAL//2NQ63lIVXHxrrS7jRyXmXUQvTPBtc2PYdvkCv15oAiAKIIX1AJ6GZmbn
coKYKkpxQYHFIS9s33UpweHQunIdCQB3AOg+0No+9QY1MudXKLyJa8kD08vREWvs
62nhd31tBr1uAAABgjwKxagAAAQDAEgwRgIhAJjY8dfyDTwnMdvkpQQecFFMgRIJ
/s3mdaYLbdh0IkRQAiEAjT5F9vfckv8llfCXjUzVBPSvqu3aLSXVklpkHHgGX4sw
DQYJKoZIhvcNAQELBQADggEBABk0SsR02kYtjwI1v9IId8htcmA+zyzqGOIWwvsj
10FGzi8BoRbmA7KiT+Yvh2CFuReOj8RNerv/BrS+PVgbHiwkh8GP2ETq8OPiXlUP
6hfSVyK5EMZBZ8Gn+JVCWrD6aUae7W53iFdZB1Bn1c4xTdBJy3eu/aYLtYHPH24p
3MQO24HcCGkn/LXGq005RMNdlTPJV/tymae2iUp5nGOy8MNkZq8xYHkK+YKue8mG
WvWdkdJ3fmYeKCXDVtvwKGk/1BQaS9lS50JLJTDJqAH957cvpyVnyOs41g19LwkZ
6MWFGfWo6Pem+d2GqZUs3faIFqmCS0lp7vHJ6e9nBuPi2do=
-----END CERTIFICATE-----
subject=CN = pypi.org

issuer=C = BE, O = GlobalSign nv-sa, CN = GlobalSign Atlas R3 DV TLS CA 2022 Q3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3502 bytes and written 398 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 5657E0EF75EFD0F949C5C55858FF62B79E65FDC5F0B8C6F47B9A57CF9B9C4B69
    Session-ID-ctx: 
    Master-Key: 34D59D972806B0540FAC90D55895AC8A16CA446DA35D2F5E3147FAD1282C25DE03187E1B72FF875D4370D830C81785D3
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - dd cd 55 d5 a5 13 4d 2a-7a 58 dd a7 df e2 f2 13   ..U...M*zX......
    0010 - 29 39 6e 1b 73 23 73 6e-28 3f f1 a7 ef 33 9c ca   )9n.s#sn(?...3..
    0020 - 66 16 be 4e f5 21 80 aa-fe d7 cb ef 56 a4 20 2e   f..N.!......V. .
    0030 - 0d 69 ac e4 66 6a 32 42-1e f5 c0 84 18 66 8a 23   .i..fj2B.....f.#
    0040 - f0 03 0d 66 66 04 d2 85-b3 f8 3f 53 d7 be 00 1c   ...ff.....?S....
    0050 - 4b 39 f3 9c 1d 50 a2 04-34 da 12 29 a0 d7 0a 30   K9...P..4..)...0
    0060 - 15 d5 d9 88 aa bc fc bb-0d c3 67 37 9d f0 00 9e   ..........g7....
    0070 - 56 31 c2 04 50 61 a5 95-fe a0 51 1e 93 90 71 dc   V1..Pa....Q...q.
    0080 - b6 20 67 5d 5a b5 ce 09-2b 65 cf 47 ad 5d d3 6c   . g]Z...+e.G.].l
    0090 - b9 00 34 09 dc 91 04 c7-b2 37 65 1c 4a a4 56 6f   ..4......7e.J.Vo
    00a0 - d9 d1 42 56 15 4c 70 db-b0 31 e4 34 3b b0 07 5d   ..BV.Lp..1.4;..]
    00b0 - 03 ac 22 dd 13 26 4f 3e-3c 46 4e 7a 52 89 c1 a6   .."..&O><FNzR...

    Start Time: 1666294258
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---

TLS Debug / IPv6 (If available)

$ echo -n | openssl s_client -6 -connect pypi.org:443

$ echo -n | openssl s_client -6 -connect files.pythonhosted.org:443

Code of Conduct

cw00dw0rd commented 1 year ago

I am experiencing the same issue using GitHub Actions against test.pypi Job Output: https://github.com/arangoml/arangodb_datasets/actions/runs/3293137897/jobs/5429395183

ewdurbin commented 1 year ago

test.pypi.org service restored.