search

pypi / legacy

This PyPI is no more! See https://github.com/pypa/warehouse.
Other
61 stars 46 forks source link

Backwards-compatible un-hashed package paths #438

Closed ewdurbin closed 8 years ago

ewdurbin commented 8 years ago

Originally reported by: Johannes Löthberg (Bitbucket: kyrias, GitHub: kyrias)

Hello,

For Arch Linux packages we very often use PyPI URLs to download the tarballs of software. The old process for updating was generally just changing the package version and updating the hash, and it would Just Work™. But with the new hashed package paths we need to go to the PyPI website manually and copy the URL to be able to update. This gets really annoying when having to update a lot of python packages.

Is there any chance that you could add a new URL path that lets you get all files without having to specify the hash? I understand why it was added, but it's useless for us since we check the checksum of the file ourselves, and just makes the process more annoying.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

Oh, and to further prove my point. I just spun up a Fedora 23 box, observed py2pack broken, replaced http://python.org/pypi (which hasn't worked in... well even longer than 3 years afaik) in it's XMLRPC call with https://pypi.python.org/pypi and both fetch and generate appears to work perfectly fine. So yea, not even a little bit related.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

First of all, the py2pack error does not appear to be related to this change at all, at least the only error I can find in any of the relevant bug trackers is https://bugzilla.redhat.com/show_bug.cgi?id=1229898 which is entirely about the XMLRPC API URL not this URL and that change was made in like... 2013? To redirect all HTTP traffic to HTTPS. If there's another bug that you'd like to point me to, by all means be my guest, but as far as I can tell py2pack does not make any assumptions about the structure of the URL and in fact it used the proper APIs to generate the .spec files.

Second of all, the thing is absolutely a race condition, it just so happens setuptools triggers it pretty easily because it has a near constant request rate and regularly releases. It also so happens I spent the last few hours sorting out the race condition and got it mostly fixed. The only vestiges left are for folks publishing to Warehouse today (rather than to PyPI) which will be fixed later this week.

ewdurbin commented 8 years ago

Original comment by Sean Farley (Bitbucket: seanfarley, GitHub: seanfarley):

Nico nailed it. And the frequency of Yen's 404 is why I deem this a:

train_wreck.jpg

ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

https://pypi.io/packages/source/s/setuptools/setuptools-22.0.0.tar.gz gives 404 as usual. The frequency is much higher than "race conditions". Can pypi.io and pypi.python.org get merged soon? Or we can ask pypa/setuptools to upload files to pypi.io?

ewdurbin commented 8 years ago

Original comment by Nico Kadel-Garcia (Bitbucket: nkadel, GitHub: nkadel):

Heads up: Every deployed version of py2pack in every RHEL and Fedora environment is now broken because of these changes and needs to be updated to operate at all. Backporting the changes is complicated because of the dependencies for py2pack: you cannot run a reliable, stable server environment and just run "pip install" at whim.

I know it's common for some Python developers, but it can cause real chaos to just "pip install the latest version" of anything, especially with the sphinx incompatibilities among different components which may be locked at previous sphinx releases.

ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

Much thanks for the explanation.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

No-- files.pythonhosted.org is part of the new site (pypi.io) but people still largely upload to the old code base (pypi.python.org) and the two don't know how to purge each other's caches.

ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

Thanks.

a race condition between the old and new site

Did you mean there's an old endpoint that can be used?

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

It's a cache issue due to a race condition between the old and new site. Setuptools is popular enough that it hits the race condition fairly regularly so I just purged the bad cache.

ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

Have no idea why it works immediately after I complain here...

ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

setuptools-21.2.1.tar.gz is uploaded yesterday and available at https://pypi.python.org/packages/13/e8/35d9c7528b3c266a17e888bea1e02eb061e9ab6cdabc7107dfb7da83a1d2/setuptools-21.2.1.tar.gz#md5=b6f59b1987fe9642874448e54ee33315. But the API endpoint gives 404.

$ curl -v -L https://pypi.io/packages/source/s/setuptools/setuptools-21.2.1.tar.gz > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 151.101.128.223...
* Connected to pypi.io (151.101.128.223) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [113 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3384 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=Delaware; serialNumber=3359300; street=16 Allen Rd; postalCode=03894-4801; C=US; ST=NH; L=Wolfeboro,; O=Python Software Foundation; CN=www.python.org
*  start date: Sep  5 00:00:00 2014 GMT
*  expire date: Sep  9 12:00:00 2016 GMT
*  subjectAltName: host "pypi.io" matched cert's "pypi.io"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
*  SSL certificate verify ok.
} [5 bytes data]
> GET /packages/source/s/setuptools/setuptools-21.2.1.tar.gz HTTP/1.1
> Host: pypi.io
> User-Agent: curl/7.48.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/1.1 301 Moved Permanently
< Location: https://files.pythonhosted.org/packages/source/s/setuptools/setuptools-21.2.1.tar.gz
< Content-Security-Policy: base-uri 'self'; block-all-mixed-content; connect-src 'self'; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.herokuapp.com/ https://secure.gravatar.com; referrer origin-when-cross-origin; reflected-xss block; script-src 'self'; style-src 'self' fonts.googleapis.com
< Content-Type: text/html; charset=UTF-8
< Fastly-Debug-Digest: 25e71b21bf2c699183325ec2d32e495199f89689c41402b983cd41f1b94bb4bd
< Content-Length: 281
< Accept-Ranges: bytes
< Date: Mon, 23 May 2016 09:39:55 GMT
< Age: 0
< Connection: keep-alive
< X-Served-By: cache-iad2125-IAD, cache-sjc3637-SJC
< X-Cache: MISS, MISS
< X-Cache-Hits: 0, 0
< X-Timer: S1463996395.564829,VS0,VE116
< Vary: Accept-Encoding
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Permitted-Cross-Domain-Policies: none
< 
* Ignoring the response-body
{ [281 bytes data]
100   281  100   281    0     0    344      0 --:--:-- --:--:-- --:--:--   344
* Connection #0 to host pypi.io left intact
* Issue another request to this URL: 'https://files.pythonhosted.org/packages/source/s/setuptools/setuptools-21.2.1.tar.gz'
*   Trying 172.111.99.6...
* Connected to files.pythonhosted.org (172.111.99.6) port 443 (#1)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [113 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3266 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc.; CN=o.ssl.fastly.net
*  start date: May 20 23:44:04 2016 GMT
*  expire date: Dec 28 14:42:27 2018 GMT
*  subjectAltName: host "files.pythonhosted.org" matched cert's "*.pythonhosted.org"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign CloudSSL CA - SHA256 - G3
*  SSL certificate verify ok.
} [5 bytes data]
> GET /packages/source/s/setuptools/setuptools-21.2.1.tar.gz HTTP/1.1
> Host: files.pythonhosted.org
> User-Agent: curl/7.48.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/1.1 404 Not Found
< Fastly-Debug-Digest: a5d8d4a6f771be36cb16195d6015080d8d928ee49a00a023d2cf92a61f86abd6
< Content-Length: 0
< Accept-Ranges: bytes
< Date: Mon, 23 May 2016 09:39:57 GMT
< Age: 0
< Connection: keep-alive
< X-Served-By: cache-iad2134-IAD, cache-hkg6825-HKG
< X-Cache: MISS, MISS
< X-Cache-Hits: 0, 0
< X-Timer: S1463996395.987487,VS0,VE1073
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Permitted-Cross-Domain-Policies: none
< 
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
* Connection #1 to host files.pythonhosted.org left intact
ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

Now it's working.

ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

Dear @dstufft the tarball is now available via https://pypi.python.org/packages/8a/c1/d2de34cfd207a5c14efda8a5cd4fe7fd47ca9d91549e0a8309a15cedfbfb/setuptools-21.2.0.tar.gz#md5=1c5521ad3fce974ff136e89a71190f60 but the API still gives 404. Should I wait some time for syncing or something?

$ curl -v -L https://pypi.io/packages/source/s/setuptools/setuptools-21.2.0.tar.gz > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 151.101.0.223...
* Connected to pypi.io (151.101.0.223) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [113 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3384 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=Delaware; serialNumber=3359300; street=16 Allen Rd; postalCode=03894-4801; C=US; ST=NH; L=Wolfeboro,; O=Python Software Foundation; CN=www.python.org
*  start date: Sep  5 00:00:00 2014 GMT
*  expire date: Sep  9 12:00:00 2016 GMT
*  subjectAltName: host "pypi.io" matched cert's "pypi.io"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
*  SSL certificate verify ok.
} [5 bytes data]
> GET /packages/source/s/setuptools/setuptools-21.2.0.tar.gz HTTP/1.1
> Host: pypi.io
> User-Agent: curl/7.48.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/1.1 301 Moved Permanently
< Location: https://files.pythonhosted.org/packages/source/s/setuptools/setuptools-21.2.0.tar.gz
< Content-Security-Policy: base-uri 'self'; block-all-mixed-content; connect-src 'self'; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.herokuapp.com/ https://secure.gravatar.com; referrer origin-when-cross-origin; reflected-xss block; script-src 'self'; style-src 'self' fonts.googleapis.com
< Content-Type: text/html; charset=UTF-8
< Fastly-Debug-Digest: fe5843c85b16537ac1cefa90637843b51123144583cdcbd052345c1fc78cb972
< Content-Length: 281
< Accept-Ranges: bytes
< Date: Sun, 22 May 2016 15:52:35 GMT
< Age: 0
< Connection: keep-alive
< X-Served-By: cache-iad2126-IAD, cache-sjc3634-SJC
< X-Cache: MISS, MISS
< X-Cache-Hits: 0, 0
< X-Timer: S1463932355.309544,VS0,VE109
< Vary: Accept-Encoding
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Permitted-Cross-Domain-Policies: none
< 
* Ignoring the response-body
{ [281 bytes data]
100   281  100   281    0     0    343      0 --:--:-- --:--:-- --:--:--   343
* Connection #0 to host pypi.io left intact
* Issue another request to this URL: 'https://files.pythonhosted.org/packages/source/s/setuptools/setuptools-21.2.0.tar.gz'
*   Trying 172.111.99.6...
* Connected to files.pythonhosted.org (172.111.99.6) port 443 (#1)
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [113 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3266 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc.; CN=o.ssl.fastly.net
*  start date: May 20 23:44:04 2016 GMT
*  expire date: Dec 28 14:42:27 2018 GMT
*  subjectAltName: host "files.pythonhosted.org" matched cert's "*.pythonhosted.org"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign CloudSSL CA - SHA256 - G3
*  SSL certificate verify ok.
} [5 bytes data]
> GET /packages/source/s/setuptools/setuptools-21.2.0.tar.gz HTTP/1.1
> Host: files.pythonhosted.org
> User-Agent: curl/7.48.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/1.1 404 Not Found
< Fastly-Debug-Digest: a220385aea7ff0c17138b6997d637a47431a188554b3560aec4fd85bdb36e939
< Content-Length: 0
< Accept-Ranges: bytes
< Date: Sun, 22 May 2016 15:52:36 GMT
< Age: 0
< Connection: keep-alive
< X-Served-By: cache-iad2138-IAD, cache-hkg6820-HKG
< X-Cache: MISS, MISS
< X-Cache-Hits: 0, 0
< X-Timer: S1463932355.710755,VS0,VE326
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Permitted-Cross-Domain-Policies: none
< 
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
* Connection #1 to host files.pythonhosted.org left intact
ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

Sorry, a missing source tarball is unexpected.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

The only file uploaded on that version of setuptools is setuptools-21.2.0-py2.py3-none-any.whl

ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

By the way, I'm in Taiwan, if that helps ;)

ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

setuptools-21.2.0.tar.gz yields 404 again:

$ curl -v -L -O https://pypi.io/packages/source/s/setuptools/setuptools-21.2.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 151.101.0.223...
* Connected to pypi.io (151.101.0.223) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [113 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3384 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: businessCategory=Private Organization; jurisdictionC=US; jurisdictionST=Delaware; serialNumber=3359300; street=16 Allen Rd; postalCode=03894-4801; C=US; ST=NH; L=Wolfeboro,; O=Python Software Foundation; CN=www.python.org
*  start date: Sep  5 00:00:00 2014 GMT
*  expire date: Sep  9 12:00:00 2016 GMT
*  subjectAltName: host "pypi.io" matched cert's "pypi.io"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 Extended Validation Server CA
*  SSL certificate verify ok.
} [5 bytes data]
> GET /packages/source/s/setuptools/setuptools-21.2.0.tar.gz HTTP/1.1
> Host: pypi.io
> User-Agent: curl/7.48.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/1.1 301 Moved Permanently
< Location: https://files.pythonhosted.org/packages/source/s/setuptools/setuptools-21.2.0.tar.gz
< Content-Security-Policy: base-uri 'self'; block-all-mixed-content; connect-src 'self'; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.herokuapp.com/ https://secure.gravatar.com; referrer origin-when-cross-origin; reflected-xss block; script-src 'self'; style-src 'self' fonts.googleapis.com
< Content-Type: text/html; charset=UTF-8
< Fastly-Debug-Digest: fe5843c85b16537ac1cefa90637843b51123144583cdcbd052345c1fc78cb972
< Content-Length: 281
< Accept-Ranges: bytes
< Date: Sun, 22 May 2016 13:46:35 GMT
< Age: 43
< Connection: keep-alive
< X-Served-By: cache-iad2136-IAD, cache-sjc3646-SJC
< X-Cache: MISS, HIT
< X-Cache-Hits: 0, 1
< X-Timer: S1463924795.097801,VS0,VE0
< Vary: Accept-Encoding
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Permitted-Cross-Domain-Policies: none
< 
* Ignoring the response-body
{ [281 bytes data]
100   281  100   281    0     0    349      0 --:--:-- --:--:-- --:--:--   349
* Connection #0 to host pypi.io left intact
* Issue another request to this URL: 'https://files.pythonhosted.org/packages/source/s/setuptools/setuptools-21.2.0.tar.gz'
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.111.99.6...
* Connected to files.pythonhosted.org (172.111.99.6) port 443 (#1)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [113 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3266 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc.; CN=o.ssl.fastly.net
*  start date: May 20 23:44:04 2016 GMT
*  expire date: Dec 28 14:42:27 2018 GMT
*  subjectAltName: host "files.pythonhosted.org" matched cert's "*.pythonhosted.org"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign CloudSSL CA - SHA256 - G3
*  SSL certificate verify ok.
} [5 bytes data]
> GET /packages/source/s/setuptools/setuptools-21.2.0.tar.gz HTTP/1.1
> Host: files.pythonhosted.org
> User-Agent: curl/7.48.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/1.1 404 Not Found
< Fastly-Debug-Digest: a220385aea7ff0c17138b6997d637a47431a188554b3560aec4fd85bdb36e939
< Content-Length: 0
< Accept-Ranges: bytes
< Date: Sun, 22 May 2016 13:46:35 GMT
< Age: 43
< Connection: keep-alive
< X-Served-By: cache-iad2141-IAD, cache-hkg6822-HKG
< X-Cache: MISS, HIT
< X-Cache-Hits: 0, 1
< X-Timer: S1463924795.393857,VS0,VE0
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< X-Frame-Options: deny
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Permitted-Cross-Domain-Policies: none
< 
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
* Connection #1 to host files.pythonhosted.org left intact

@dstufft Is this information enough?

ewdurbin commented 8 years ago

Original comment by Antony Lee (Bitbucket: anntzer, GitHub: anntzer):

Fair enough, thanks.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

A wheel is not a source release, it would be at https://files.pythonhosted.org/packages/py2.py3/m/matlab_kernel/matlab_kernel-0.9.5-py2.py3-none-any.whl.

ewdurbin commented 8 years ago

Original comment by Antony Lee (Bitbucket: anntzer, GitHub: anntzer):

Some redirections seem to fail, e.g. I cannot curl (404 error) https://pypi.io/packages/source/m/matlab_kernel/matlab_kernel-0.9.5-py2.py3-none-any.whl for https://pypi.python.org/pypi/matlab_kernel

ewdurbin commented 8 years ago

Original comment by Tomas Orsava (Bitbucket: torsava, GitHub: torsava):

@dstufft Thank you very much!

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

@torsava Aside from the only thing in life you can really count on being Death, Taxes, and that Software will always break you should be able to rely on that URL. It's basically a redirector in the form of pypi.debian.net just running on the PSF infrastructure (https://github.com/pypa/conveyor if you're interested) and it's operationally very simple to run for us. I don't have any further plans to change the URL structure or the location of the files and I don't see any reason why we would need to.

That being said, it would certainly be more of a guarantee that you'd never have to change those URLs again if you ran your own redirector since it'd give you a single location that you'd need to change if something ever did change... but I don't feel like there's any pressing need to do so if the redirect there is good enough for you (It's not for Debian, they wanted more than just a hard coded file).

ewdurbin commented 8 years ago

Original comment by Tomas Orsava (Bitbucket: torsava, GitHub: torsava):

Hi, @dstufft!

I want to ask you if we can count on the longevity of the https://files.pythonhosted.org/packages/source/p/positional/positional-1.1.0.tar.gz URL scheme, or if perhaps we in Fedora/RHEL need to resort to our own URL redirect in the style of pypi.debian.net.

This URL change has just broken several thousand Fedora/RHEL packages and the source links will have to be changed in every single one of them, sooner or later. Therefore it would be great if we didn't have to do the same thing again in the foreseeable future.

Thank you.

ewdurbin commented 8 years ago

Original comment by Tomas Orsava (Bitbucket: torsava, GitHub: torsava):

@andrew_shadura There already is a URL redirector, though @dstufft could have highlighted it more:

Use this format from now on:

https://files.pythonhosted.org/packages/source/p/positional/positional-1.1.0.tar.gz

However, I agree that providing it at the old location would have been wiser.

ewdurbin commented 8 years ago

Original comment by Andrew Shadura (Bitbucket: andrew_shadura, GitHub: Unknown):

Well, if I broke download links for everyone else, I probably would.

ewdurbin commented 8 years ago

Original comment by Chris Warrick (Bitbucket: Kwpolska, GitHub: Kwpolska):

@andrew_shadura Yes, the current codebase is quite hard to maintain. But why not contribute a patch on your own, help out the PyPI devs, and make everyone else in the world happy?

ewdurbin commented 8 years ago

Original comment by Andrew Shadura (Bitbucket: andrew_shadura, GitHub: Unknown):

I seem to really miss the point of whole this conversation. Is it that difficult to provide an URL redirector at the old location so that everyone's not forced to do changes on their side?

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

@yan12125 Probably the 404 got cached.

ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

It's working for me now as well - tried the same wget command 20 times as well some curl commands, none of which returns 404.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

@yan12125 It seems to be working for me, can you get the headers of the responses so I can see what cache nodes are being hit?

ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

Is it broken again? pip-8.1.2 raises a 404 error:

$ wget "https://pypi.io/packages/source/p/pip/pip-8.1.2.tar.gz"
--2016-05-11 20:18:10-- https://pypi.io/packages/source/p/pip/pip-8.1.2.tar.gz Resolving pypi.io (pypi.io)... 151.101.0.223, 151.101.192.223, 151.101.128.223, ... Connecting to pypi.io (pypi.io)|151.101.0.223|:443... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://files.pythonhosted.org/packages/source/p/pip/pip-8.1.2.tar.gz [following] --2016-05-11 20:18:11-- https://files.pythonhosted.org/packages/source/p/pip/pip-8.1.2.tar.gz Resolving files.pythonhosted.org (files.pythonhosted.org)... 172.111.99.6 Connecting to files.pythonhosted.org (files.pythonhosted.org)|172.111.99.6|:443... connected. HTTP request sent, awaiting response... 404 Not Found 2016-05-11 20:18:12 ERROR 404: Not Found.

While pip 8.1.1 works: $ wget "https://pypi.io/packages/source/p/pip/pip-8.1.1.tar.gz" --2016-05-11 20:18:20-- https://pypi.io/packages/source/p/pip/pip-8.1.1.tar.gz Resolving pypi.io (pypi.io)... 151.101.192.223, 151.101.64.223, 151.101.128.223, ... Connecting to pypi.io (pypi.io)|151.101.192.223|:443... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://files.pythonhosted.org/packages/source/p/pip/pip-8.1.1.tar.gz [following] --2016-05-11 20:18:20-- https://files.pythonhosted.org/packages/source/p/pip/pip-8.1.1.tar.gz Resolving files.pythonhosted.org (files.pythonhosted.org)... 172.111.99.6 Connecting to files.pythonhosted.org (files.pythonhosted.org)|172.111.99.6|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://files.pythonhosted.org/packages/41/27/9a8d24e1b55bd8c85e4d022da2922cb206f183e2d18fee4e320c9547e751/pip-8.1.1.tar.gz [following] --2016-05-11 20:18:21-- https://files.pythonhosted.org/packages/41/27/9a8d24e1b55bd8c85e4d022da2922cb206f183e2d18fee4e320c9547e751/pip-8.1.1.tar.gz Reusing existing connection to files.pythonhosted.org:443. HTTP request sent, awaiting response... 200 OK Length: 1139175 (1.1M) [application/octet-stream] Saving to: ‘pip-8.1.1.tar.gz’

pip-8.1.1.tar.gz 100%[====================================================================>] 1.09M 5.94MB/s in 0.2s

2016-05-11 20:18:22 (5.94 MB/s) - ‘pip-8.1.1.tar.gz’ saved [1139175/1139175]

ewdurbin commented 8 years ago

Original comment by Nico Kadel-Garcia (Bitbucket: nkadel, GitHub: nkadel):

I really wish you'd not try to solve the "people replace their own tarballs" problem with "let's write clever scripting to change the URL's, no one will notice". We noticed, and the new structure has just demonstrated its fragility, again. Even if the new front end is stable, it's extra traffic and complexity that is protecting eager python users from ever eager developers who might release mislabeled code.

Does anyone actually want the old version when distinct versions of the same tarball are published? Wouldn't it be safer to lock tarballs upon upload, and compel the developer to update their tarball number to publish a new one, as a warning to be more cautious? I'm not sure what your backend source control is, or if you use one to pre-stage uploads to S3. But tools can be configured to remove write privileges after a tarball is uploaded, and a really bad tarball should probably be deleted from the public repo and never returned, rather than providing dual access to both old and new tarballs. A numbered tarball, unless it's an internal-only dev version, should be canonical.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

@workmanw should be fixed, was a bug in some code I pushed last night.

ewdurbin commented 8 years ago

Original comment by Wesley Workman (Bitbucket: workmanw, GitHub: workmanw):

So for the 3rd time in two weeks my build system is completely broken because of these changes. I've tried very hard to be patient and understanding because I've been where you are. This link, which has worked numerous times and I've cited in other responses, no longer is functioning.

https://pypi.io/packages/source/s/setuptools/setuptools-20.10.1.zip

I've made code changes to our repo, I've reported issues here. I don't know what more to do to ensure stability in our build pipeline. If you have and ideas, I'm all ears.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

Just so people who are following this knows, I've further adjusted how URLs are served on Warehouse. All files are now hosted on a separate domain (to prevent things like giffar attacks) but the URL structure on that domain stays the same (in addition (test.)pypi.io/packages/* will redirect to the new URLs. So if someone has switched to using pypi.io already nothing should break, but if you're using something like https://pypi.io/packages/source/p/positional/positional-1.1.0.tar.gz as you're URL you're going to go through 3 HTTP requests to actually get the file (Once to redirect to https://files.pythonhosted.org/packages/source/p/positional/positional-1.1.0.tar.gz, once to redirect to https://files.pythonhosted.org/pacakges/bf/a8/bc656a556a60b76c32830b57279f51714ab7c6366fd243d6ea86b6fcad46/positional-1.1.0.tar.gz and once to download the actual file. It is suggested that people switch to using files.pythonhosted.org directly, but the pypi.io URLs will not go away.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

@asavah Regardless of how you may feel about the change, please refrain from personal attacks, particularly ones that put down a whole group of people.

ewdurbin commented 8 years ago

Original comment by asavah (Bitbucket: asavah, GitHub: asavah):

I understand the reasons of such a change, but whoever did this without a previous announcement and a mechanism providing old behaviour DID break a lot of build systems which relied on the old good urls for package downloads, I appreciate you work, but consider this: you wake up in the morning and get into your car to drive to work but there is no more steering wheel, but some alien device to control the car because your car vendor decided to change UI overnight. Your reaction? I'd just call that vendor a retarded moron, sorry, but that's about you too.

ewdurbin commented 8 years ago

Original comment by Bernard Spil (Bitbucket: Barnerd, GitHub: Barnerd):

There's been an upstream change making pypi.io/packages return 302 in stead of 307 which allows us to solve the issue with FreeBSD ports.

That doesn't stop us from implementing 307 support as well, as should you...

See: https://github.com/pypa/warehouse/pull/1165

#!shell
fetch -v https://pypi.io/packages/source/p/positional/positional-1.1.0.tar.gz
302 redirect to https://pypi.io/packages/bf/a8/bc656a556a60b76c32830b57279f51714ab7c6366fd243d6ea86b6fcad46/positional-1.1.0.tar.gz
$ fetch https://pypi.io/packages/source/p/positional/positional-1.1.0.tar.gz
positional-1.1.0.tar.gz                       100% of   15 kB   82 MBps 00m00s
ewdurbin commented 8 years ago

Original comment by Kenneth Hoste (Bitbucket: kehoste, GitHub: Unknown):

Putting this here for future reference (copied from #python IRC channel):

Thanks @dstufft for the info.

ewdurbin commented 8 years ago

Original comment by Wesley Workman (Bitbucket: workmanw, GitHub: workmanw):

@dstufft @The-Compiler I can confirm that the warehouse functionality is restore.

ewdurbin commented 8 years ago

Original comment by Kenneth Hoste (Bitbucket: kehoste, GitHub: Unknown):

@dstufft That's a great way to serve the community. Thanks.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

Patches Accepted.

ewdurbin commented 8 years ago

Original comment by Kenneth Hoste (Bitbucket: kehoste, GitHub: Unknown):

Just when you thought Python packaging couldn't be made any worse...

So, let me get this right... The download URL on the production PyPI was changed for a particular reason (fine), without putting a workaround in place for tools that were relying on the pretty straightforward easily derivable URL that files were being served on for years.

And then on pypi.io, the pre-production, a workaround was put in place by mirroring the old-style URL to the new-style URL. But, this workaround isn't going to be 'ported' to the current production PyPI?

What's the use of a pre-production setup if you're happily changing the production setup, and telling users to just deal with what breaks?

Also: please don't underestimate how this is going to affect the Python community, especially people not very familiar with the mess that Python packaging is already.

I strongly suggest to reconsider not fixing this in the production PyPI.

ewdurbin commented 8 years ago

Original comment by Florian Bruhin (Bitbucket: The-Compiler, GitHub: The-Compiler):

FWIW using curl with --compressed (or the Accept-Encoding: gzip, deflate HTTP header) works

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

Should be fixed now.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

It's an unrelated breakage on Warehouse. All non browser curl requests are currently 404ing on pypi.io.

ewdurbin commented 8 years ago

Original comment by Wesley Workman (Bitbucket: workmanw, GitHub: workmanw):

Today, once again, our ability to use buildout / ez_setup is busted. It appears that ez_setup updated to a new version of setuptools. We are no longer able to cURL for our packages. The new links for old packages seem to work fine. The new links for new packages to do not. Example:

#!bash

# Old package - New style link - WORKS
$ curl 'https://pypi.io/packages/source/s/setuptools/setuptools-20.10.1.zip'

307 Temporary Redirect

The resource has been moved to /packages/86/ee/622e83b0dbede6d48891ed209fa5ca83fcc485f9b6696cf56796eda40806/setuptools-20.10.1.zip; you should be redirected automatically.

# New package - New style link - DOES NOT WORK
$ curl 'https://pypi.io/packages/source/s/setuptools/setuptools-21.0.0.zip'

<html>
 <head>
  <title>404 Not Found</title>
 </head>
 <body>
  <h1>404 Not Found</h1>
  The resource could not be found.<br/><br/>
 </body>

Interestingly enough if you use the new package link in the browser, https://pypi.io/packages/source/s/setuptools/setuptools-21.0.0.zip, it does download fine. Perhaps there are some additional HTTP header requirements imposed now to use cURL.

Any thoughts?

ewdurbin commented 8 years ago

Original comment by Chi Hsuan Yen (Bitbucket: yan12125, GitHub: yan12125):

Issue #447 was marked as a duplicate of this issue.

ewdurbin commented 8 years ago

Original comment by Michael Sarahan (Bitbucket: msarahan, GitHub: msarahan):

@dstufft confirmed as fixed over here - thank you.

ewdurbin commented 8 years ago

Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft):

@seanfarley It'll be fine. We've served half a billion requests since the change went live and breakage has been minimal. What has broken has had a mechanism to keep working implemented.