SSL Error (certificate verify failed) for Poetry `install`, `add`, etc.

KyleKing commented 5 years ago

[X] I am on the latest Poetry version.
- Just reinstalled a fresh version with get_poetry.py using Python 3.7.1 (Logs: install_poetry.txt)
[X] I have searched the issues of this repo and believe that this is not a duplicate.
- None of the similar issues were applicable: #680, #449, and #165
[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).
OS version and name: Windows 7
Poetry version: 0.12.10
Contents of your pyproject.toml file: pyproject.toml (*renamed to .txt to upload)
- This is the just the default from poetry new hello_world with my email address obscured

Issue

How can I point Poetry to a custom .pem certificate?

Poetry isn't using the correct certificate when accessing pypi.org. For pip, conda, and other applications I have had to add a custom .pem file since my company uses a self-signed certificate. The pip in the virtualenv created by Poetry use the correct setting (see below: C:\\Programs\\cert\\cacert.pem), but Poetry throws an SSLError when running any action

C:\Users\king.kyle\hello_world>poetry -vvv install
Using virtualenv: C:\Users\king.kyle\AppData\Local\pypoetry\Cache\virtualenvs\hello-world-py3.7
Updating dependencies
Resolving dependencies...
   1: fact: hello-world is 0.1.0
   1: derived: hello-world
   1: fact: hello-world depends on pytest (^3.0)
   1: selecting hello-world (0.1.0)
   1: derived: pytest (^3.0)
   1: Version solving took 0.053 seconds.
   1: Tried 1 solutions.

[SSLError]
HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/pytest/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verif
y failed: self signed certificate in certificate chain (_ssl.c:1051)')))

Exception trace:
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cleo\application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\console\application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cleo\application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\console\commands\command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cleo\commands\base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cleo\commands\command.py in execute() at line 107
   return self.handle()
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\console\commands\install.py in handle() at line 57
   return_code = installer.run()
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\installation\installer.py in run() at line 76
   self._do_install(local_repo)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\installation\installer.py in _do_install() at line 158
   ops = solver.solve(use_latest=self._whitelist)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\puzzle\solver.py in solve() at line 38
   packages, depths = self._solve(use_latest=use_latest)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\puzzle\solver.py in _solve() at line 171
   self._package, self._provider, locked=locked, use_latest=use_latest
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\mixology\__init__.py in resolve_version() at line 7
   return solver.solve()
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\mixology\version_solver.py in solve() at line 79
   next = self._choose_package_version()
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\mixology\version_solver.py in _choose_package_version() at line 354
   packages = self._provider.search_for(dependency)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\puzzle\provider.py in search_for() at line 146
   allow_prereleases=dependency.allows_prereleases(),
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pool.py in find_packages() at line 65
   name, constraint, extras=extras, allow_prereleases=allow_prereleases
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pypi_repository.py in find_packages() at line 104
   info = self.get_package_info(name)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pypi_repository.py in get_package_info() at line 228
   name, lambda: self._get_package_info(name)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cachy\repository.py in remember_forever() at line 174
   val = value(callback)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cachy\helpers.py in value() at line 6
   return val()
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pypi_repository.py in <lambda>() at line 228
   name, lambda: self._get_package_info(name)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pypi_repository.py in _get_package_info() at line 232
   data = self._get("pypi/{}/json".format(name))
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pypi_repository.py in _get() at line 381
   json_response = self._session.get(self._url + endpoint)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\requests\sessions.py in get() at line 546
   return self.request('GET', url, **kwargs)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\requests\sessions.py in request() at line 533
   resp = self.send(prep, **send_kwargs)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\requests\sessions.py in send() at line 646
   r = adapter.send(request, **kwargs)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cachecontrol\adapter.py in send() at line 53
   resp = super(CacheControlAdapter, self).send(request, **kw)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\requests\adapters.py in send() at line 514
   raise SSLError(e, request=request)

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

(py37) C:\Users\king.kyle\hello_world>

Code

Here are the abbreviated logs from running the minimum steps to reproduce:

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\king.kyle>poetry new hello_world
Created package hello_world in hello_world

C:\Users\king.kyle>cd hello_world

C:\Users\king.kyle\hello_world>poetry install
Creating virtualenv hello-world-py3.7 in C:\Users\king.kyle\AppData\Local\pypoetry\Cache\virtualenvs
Updating dependencies
Resolving dependencies...

[SSLError]
HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/pytest/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verif
y failed: self signed certificate in certificate chain (_ssl.c:1051)')))

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

C:\Users\king.kyle\hello_world>poetry shell
Spawning shell within C:\Users\king.kyle\AppData\Local\pypoetry\Cache\virtualenvs\hello-world-py3.7
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.

C:\Users\king.kyle\hello_world>pip install requests
Collecting requests
  Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting urllib3<1.25,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl (154kB)
    100% |¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦| 163kB 6.8MB/s
Collecting idna<2.9,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: chardet, urllib3, certifi, idna, requests
Successfully installed certifi-2018.11.29 chardet-3.0.4 idna-2.8 requests-2.21.0 urllib3-1.24.1
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

C:\Users\king.kyle\hello_world>poetry -vvv add requests
Using virtualenv: C:\Users\king.kyle\AppData\Local\pypoetry\Cache\virtualenvs\hello-world-py3.7

[SSLError]
HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/requests/json (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate ver
ify failed: self signed certificate in certificate chain (_ssl.c:1051)')))

Exception trace:
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cleo\application.py in run() at line 94
   status_code = self.do_run(input_, output_)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\console\application.py in do_run() at line 88
   return super(Application, self).do_run(i, o)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cleo\application.py in do_run() at line 197
   status_code = command.run(input_, output_)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\console\commands\command.py in run() at line 77
   return super(BaseCommand, self).run(i, o)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cleo\commands\base_command.py in run() at line 146
   status_code = self.execute(input_, output_)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cleo\commands\command.py in execute() at line 107
   return self.handle()
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\console\commands\add.py in handle() at line 69
   packages, allow_prereleases=self.option("allow-prereleases")
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\console\commands\init.py in _determine_requirements() at line 230
   requirement["name"], allow_prereleases=allow_prereleases
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\console\commands\init.py in _find_best_version_for_package() at line 260
   name, required_version, allow_prereleases=allow_prereleases
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\version\version_selector.py in find_best_candidate() at line 29
   package_name, constraint, allow_prereleases=allow_prereleases
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pool.py in find_packages() at line 65
   name, constraint, extras=extras, allow_prereleases=allow_prereleases
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pypi_repository.py in find_packages() at line 104
   info = self.get_package_info(name)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pypi_repository.py in get_package_info() at line 228
   name, lambda: self._get_package_info(name)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cachy\repository.py in remember_forever() at line 174
   val = value(callback)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cachy\helpers.py in value() at line 6
   return val()
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pypi_repository.py in <lambda>() at line 228
   name, lambda: self._get_package_info(name)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pypi_repository.py in _get_package_info() at line 232
   data = self._get("pypi/{}/json".format(name))
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\poetry\repositories\pypi_repository.py in _get() at line 381
   json_response = self._session.get(self._url + endpoint)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\requests\sessions.py in get() at line 546
   return self.request('GET', url, **kwargs)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\requests\sessions.py in request() at line 533
   resp = self.send(prep, **send_kwargs)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\requests\sessions.py in send() at line 646
   r = adapter.send(request, **kwargs)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\cachecontrol\adapter.py in send() at line 53
   resp = super(CacheControlAdapter, self).send(request, **kw)
 c:\users\king.kyle\appdata\local\continuum\anaconda2\envs\py37\lib\site-packages\requests\adapters.py in send() at line 514
   raise SSLError(e, request=request)

add [-D|--dev] [--git GIT] [--path PATH] [-E|--extras EXTRAS] [--optional] [--python PYTHON] [--platform PLATFORM] [--allow-prereleases] [--dry-run] [--] <name> (<name>)...

C:\Users\king.kyle\hello_world>python -m pip config list
global.cert='C:\\Programs\\cert\\cacert.pem'
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

C:\Users\king.kyle\hello_world>

KyleKing commented 5 years ago

To side-step the certificate issue. Would it be possible to point to a shared directory? This doesn't work now because file:// isn't a recognized adapter:

[[tool.poetry.source]]
name = "M"
url = "file://corporate.com/dfs/PythonPackages/"

Edit: created new ticket for this suggestion: #748

KyleKing commented 5 years ago

Turns out requests recognize an environment variable REQUESTS_CA_BUNDLE https://github.com/requests/requests/blob/master/requests/sessions.py#L707

Once set, Poetry worked without issue. Hopefully, this helps someone else out!

Edit: Also, Jean-Philippe summarizes this issue perfectly: https://bugs.python.org/issue28547

KyleKing commented 4 years ago

See https://github.com/python-poetry/poetry/issues/1012#issuecomment-567029119 for a solution using the new cert configuration item. This can likely replace the need for the REQUESTS_CA_BUNDLE environment variable

tchaton commented 4 years ago

Same trouble this morning

[EnvCommandError]
Command ['/Users/tcn02/projects/perso/lightning-hydra-seed/.venv/bin/pip', 'install', '--no-deps', 'black==20.8b1'] errored with the following return code 1, and output: 
Collecting black==20.8b1
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)'))': /packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)'))': /packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)'))': /packages/dc/7b/5a6bbe89de849f28d7c109f5ea87b65afa5124ad615f3419e71beb29dc96/black-20.8b1.tar.gz

KyleKing commented 4 years ago

@tchaton you may want to open a new issue. I tested poetry latest with my network’s self-signed certificates yesterday and the recommended fixes still work. In your case, it looks like something else is happening (not a self-signed certificate error, at least not from your posted error message)

I would recommend posting more information about your IT environment, poetry and OS versions, and the steps you can take to replicate the failure? Does pip install work with poetry run pip install. Is it only black that fails to install or all packages, etc.

kryptoatom commented 3 years ago

Below steps worked for me.

Connect to any https site with browser
Click on lock icon just to left of https://
View and save root certificate

Convert cert to .pem I used openssl that comes with git.

# cd C:\Users\<my_username>\AppData\Local\Programs\Git\usr\bin>
# ./openssl x509 -in "C:\Users\<my_username>\<my_folder>\my_cert.cer" -out "C:\Users\<my_username>\<my_folder>\my_cert.pem"

Open my_cert.pem in a notepad/notepad++. Copy the text
Locate your cacert.pem. In my case it was in path below. C:\Users\<my_username>\.poetry\lib\poetry\_vendor\py3.9\certifi\cacert.pem
Paste the text copied in step 5 to cacert.pem and save
You just achieved Nirvana!

danibachini commented 2 years ago

what worked for me (MacOS): go to Applications > Python folder > double click on "Install Certificates.command" file

cardosofede commented 2 years ago

@danibachini you are the queen!

aaroncroberts commented 2 years ago

@danibachini Perfect. Thank you.

cevaris commented 1 year ago

what worked for me (MacOS): go to Applications > Python folder > double click on "Install Certificates.command" file

specifically, run this command worked for me /Applications/Python\ 3.11/Install\ Certificates.command.

raajkumars commented 1 year ago

https://github.com/requests/requests/blob/master/requests/sessions.py#L707

Thanks! This is correct solution for this issue for those Mac OS X.

cove9988 commented 10 months ago

find your_ca_bundle.pem file, because your poetry commands need SSL certifications,

export REQUESTS_CA_BUNDLE=~/your_cert_path/your_ca_bundle.pem

github-actions[bot] commented 8 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

python-poetry / poetry

SSL Error (certificate verify failed) for Poetry `install`, `add`, etc. #745

Issue

Code