search

pypa / twine

Utilities for interacting with PyPI
https://twine.readthedocs.io/
Apache License 2.0
1.6k stars 308 forks source link

upload to untrusted hosts with --trusted option #387

Closed glenfant closed 4 years ago

glenfant commented 6 years ago

Hi,

Twine denies uploading to untrusted https PyPI clones. This triggers an SSL error caused by an unknown CA certificate. I need the ability to upload packages to untrusted private PyPI servers since

Backgroung : I'm in process to add PyPI and NPM support to a Nexus server that's published in production with regular CA signed certificates. But can't obtain such certificates for the Nexus test site.

I made a small quick'n'dirty patch setting verify=False option when posting a tarball or wheel file here. https://github.com/pypa/twine/blob/master/twine/repository.py#L152

I know that the requests lib issues a warning in such situations but I don't care.

Thanks

glenfant commented 5 years ago

Would you accept a PR with a new option "trusted-host" option that does the same job as for pip ?

$ pip help install
...
  --trusted-host <hostname>   Mark this host as trusted, even though it does
                              not have valid or any HTTPS.
...
sigmavirus24 commented 5 years ago

That seems like a good way to name it and go about it, @glenfant .

piersf commented 5 years ago

Hello there,

Was this ever implemented? We need this badly as setup.py upload command does not seem to be working anymore with Python 3.7.x versions. when uploading to private PyPi repositories(gives 400 Bad Request)

bhrutledge commented 5 years ago

Thanks for the ping, @piersf. There's currently an open PR #463 that's waiting for the submitter to make some changes. I'll ping him to see if/when he's able to do that.

di commented 5 years ago

Sorry I didn't comment earlier, but I wanted to say that I think I'm -1 on making this change.

I'm a bit concerned that if we add this flag, it will proliferate through comments, stack overflow answers, etc as an "easy" way to fix SSL errors that should actually be fixed by producing and obtaining valid certificates for third-party indices. Basically, this shouldn't be necessary and feels like an easy way out.

I'm especially concerned about the possibility of this being used with --trusted-host pypi.org which should basically never happen. At the very least, I'd like to see this being implemented in a way that prevents this flag from ever being used with that host.

(cc @ewdurbin for thoughts here as well)

glenfant commented 5 years ago

I agree, it's preferable and technically easy to add a real CA approved certificate to the private registry server (Nexus) of my customer's company, but it's a political nightmare. I can understand your opinion, but the only workaround I found is to add to every workstation, CI container (etc.) a script that add the private CA cert to the standard keyring :/

glenfant commented 5 years ago

@di If I understand correctly, could you accept a change that ignores --trusted on *.pypi.org target registry?

itaym120 commented 4 years ago

Sorry I didn't comment earlier, but I wanted to say that I think I'm -1 on making this change.

I'm a bit concerned that if we add this flag, it will proliferate through comments, stack overflow answers, etc as an "easy" way to fix SSL errors that should actually be fixed by producing and obtaining valid certificates for third-party indices. Basically, this shouldn't be necessary and feels like an easy way out.

I'm especially concerned about the possibility of this being used with --trusted-host pypi.org which should basically never happen. At the very least, I'd like to see this being implemented in a way that prevents this flag from ever being used with that host.

(cc @ewdurbin for thoughts here as well)

It's not always possible to work the "right" way, that's why pip, conda install and npm gives the option to add trusted-host\verify-ssl flags. Not as the recommended way, but as a possiblity for users. The default should obviously be to verify whenever its a possibility.

In my case, I just can't demand from the Artifactory repository providers to add a real CA approved certificate and I have no choise but use verify=False when uploading and downloading packages from there. and there are no alternatives in the lan...

jaraco commented 4 years ago

Is there a better option here? Is it possible without disabling SSL to still trust/allow it? That is, is it possible to just download the certificate and allow it? Something like:

import ssl
import sys
import urllib.parse

url, = sys.argv[1:]
if '/' not in url:
    url = '//' + url
parsed = urllib.parse.urlparse(url)

with open('cert.pem', 'w') as f:
    f.write(ssl.get_server_certificate((parsed.hostname, parsed.port or 443)))

to download the certificate and then

REQUESTS_CA_BUNDLE=cert.pem twine upload ...

According to this answer, I'd expect that to work, but it didn't work for me when I tested against self-signed.badssl.com.

sigmavirus24 commented 4 years ago

@jaraco this is what --cert is supposed to provide. If you have the certificates as a PEM file which can be generated trivially by searching StackOverflow, then they should be able to pass that to --cert iirc instead of having --trusted. That solves only the one case that @glenfant described though.

There's still the issue of "what about an implementation without TLS?" I think the answer there is "maybe that works today already." I don't think we validate for https:// as a scheme or change that forcefully. If we do, then we'd need an option along the lines of --yes-i-know-i-should-use-tls but that wouldn't be fixed by --trusted because we'd otherwise still try connecting over :443 and the server may not even listen on that port.

jaraco commented 4 years ago

I can say with confidence that upload requests over http work, because twine now has integration tests against a local devpi instance over plaintext HTTP.

okainov commented 4 years ago

+1 for the issue, I'd really like to have this option in

jaraco commented 4 years ago

The solution to this problem is threefold:

This last option is intentionally ugly and inconvenient to limit the proliferation of that discouraged approach.

If none of these approaches are suitable, please feel free to elaborate on why.

Eliav2 commented 1 year ago

This issue really shouldn't be closed . You are not special for not allowing "insecure" requests on secured isolated networks

jaraco commented 1 year ago

pip, conda install and npm gives the option to add trusted-host\verify-ssl flags

[twine is] not special for not allowing "insecure" requests

Maybe twine is special, because it's responsible for publishing whereas the other tools mentioned are primarily responsible for consuming. There's a higher standard of security demanded from tools that publish content.

Does npm also allow the same flags for publishing? Are there other examples of tools for publishing work that allow bypassing critical security checks.

I don't think anyone has yet proposed a technique that would allow disabling of the security checks when they're not needed but would disallow the same where the checks should be enforced. As a general rule, if the server is only exposing HTTPS, then it's the client's responsibility to validate the authenticity of the server. Twine would need a way to prevent casual bypass of the check. How can twine discriminate between legitimate bypass and casual bypass?

[supplying the correct cert] is what --cert is supposed to provide. If you have the certificates as a PEM file... then they should be able to pass that to --cert.

I tried this approach this time I was able to make it work using the script above as get-cert.py. 

 draft @ py -m get-cert https://self-signed.badssl.com
 draft @ http -q --download https://files.pythonhosted.org/packages/c9/c9/6122a974f5b611b16396c918722ea75945db7dcd3069c477a7c608a405a0/example-0.1.
0.tar.gz
 draft @ twine upload --password foo --repository-url https://self-signed.badssl.com/upload/ example-0.1.0.tar.gz
Uploading distributions to https://self-signed.badssl.com/upload/
Uploading example-0.1.0.tar.gz
WARNING  Retrying (Retry(total=9, connect=5, read=None, redirect=None, status=None)) after connection broken by                                    
         'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate                
         (_ssl.c:1002)'))': /upload/                                                                                                               
...
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='self-signed.badssl.com', port=443): Max retries exceeded with url: /upload/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1002)')))
 draft @ twine upload --password foo --cert cert.pem --repository-url https://self-signed.badssl.com/upload/ example-0.1.0.tar.gz
Uploading distributions to https://self-signed.badssl.com/upload/
Uploading example-0.1.0.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 kB • 00:00 • ?
WARNING  Error during upload. Retry with the --verbose option for more details.                                                                    
ERROR    HTTPError: 404 Not Found from https://self-signed.badssl.com/upload/                                                                      
         Not Found

In the first attempt, I don't pass the certificate and the request fails with an SSL error. In the second attempt, I pass the downloaded certificate and pass it to twine, authorizing that specific server (or the man in the middle attacker), and it fails with a 404 error, presumably after the TLS handshake has succeeded.

That technique should work for any isolated networks. I'll amend the recommendations.

The recommendations to address this problem:

If neither of these approaches are suitable, please feel free to elaborate on why.