search

conda / conda-lock

Lightweight lockfile for conda environments
https://conda.github.io/conda-lock/
Other
499 stars 104 forks source link

Unable to specify ssl certificate for pip-repositories repository #653

Open diego-comlet opened 5 months ago

diego-comlet commented 5 months ago

Checklist

What happened?

For the given enviorment.yaml

dependencies:
  - python=3.10
  - pip==24
  - pip:
    - docutils
pip-repositories:
  - https://internal.pypi.repo.com:8443/simple

After running conda-lock lock -f enviorment.yaml, the following issue is generated:

 Locking dependencies for ['linux-64', 'win-64']...
INFO:conda_lock.conda_solver:linux-64 using specs ['python 3.10.*', 'pip *']
Traceback (most recent call last):
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\urllib3\connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\urllib3\connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\urllib3\connectionpool.py", line 1060, in _validate_conn
    conn.connect()
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\urllib3\connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\urllib3\util\ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\urllib3\util\ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\ssl.py", line 455, in wrap_socket   
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\ssl.py", line 1042, in _create      
    self.do_handshake()
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\ssl.py", line 1320, in do_handshake 
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\requests\adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\urllib3\connectionpool.py", line 801, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\urllib3\util\retry.py", line 594, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='internal.pypi.repo.com', port=8443): Max retries exceeded with url: /simple/docutils/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Scripts\conda-lock-script.py", line 10, 
in <module>
    sys.exit(main())
             ^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\click\decorators.py", 
line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\conda_lock.py", line 1403, in lock
    lock_func(
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\conda_lock.py", line 1111, in run_lock
    make_lock_files(
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\conda_lock.py", line 393, in make_lock_files
    fresh_lock_content = create_lockfile_from_spec(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\conda_lock.py", line 836, in create_lockfile_from_spec
    deps = _solve_for_arch(
           ^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\conda_lock.py", line 760, in _solve_for_arch
    pip_deps = solve_pypi(
               ^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\pypi_solver.py", line 554, in solve_pypi
    result = s.solve(use_latest=to_update)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\_vendor\poetry\puzzle\solver.py", line 65, in solve
    packages, depths = self._solve(use_latest=use_latest)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\_vendor\poetry\puzzle\solver.py", line 233, in _solve
    result = resolve_version(
             ^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\_vendor\poetry\mixology\__init__.py", line 7, in resolve_version
    return solver.solve()
           ^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\_vendor\poetry\mixology\version_solver.py", line 84, in solve
    next = self._choose_package_version()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\_vendor\poetry\mixology\version_solver.py", line 372, in _choose_package_version
    dependency = min(*unsatisfied, key=_get_min)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\_vendor\poetry\mixology\version_solver.py", line 364, in _get_min
    len(self._provider.search_for(dependency)),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\_vendor\poetry\puzzle\provider.py", line 139, in search_for
    packages = self._pool.find_packages(dependency)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\_vendor\poetry\repositories\pool.py", line 170, in find_packages
    packages += repo.find_packages(dependency)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\_vendor\poetry\repositories\legacy_repository.py", line 264, in find_packages
    page = self._get("/{}/".format(dependency.name.replace(".", "-")))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\conda_lock\_vendor\poetry\repositories\legacy_repository.py", line 388, in _get
    response = self.session.get(url)
               ^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\requests\sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\requests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\cachecontrol\adapter.py", line 76, in send
    resp = super().send(request, stream, timeout, verify, cert, proxies)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "**redacted**\micromamba-binary\envs\conda-lock-env\Lib\site-packages\requests\adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='internal.pypi.repo.com', port=8443): Max retries exceeded with url: /simple/docutils/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')))

The conda env, is configured to ignore SSL certificates, but to not avail in this case

Conda Info

active environment : base
    active env location : **redacted**\micromamba-binary\envs\conda-lock-env
            shell level : 1
       user config file : C:\Users\**username**\.condarc
 populated config files : **redacted**\micromamba-binary\envs\conda-lock-env\.condarc
          conda version : 24.5.0
    conda-build version : not installed
         python version : 3.12.4.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=24.5.0=0
                          __cuda=12.2=0
                          __win=0=0
       base environment : **redacted**\micromamba-binary\envs\conda-lock-env  (writable)
      conda av data dir : **redacted**\micromamba-binary\envs\conda-lock-env\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : **redacted**\micromamba-binary\envs\conda-lock-env\pkgs
                          C:\Users\**username**\.conda\pkgs
                          C:\Users\**username**\AppData\Local\conda\conda\pkgs
       envs directories : **redacted**\micromamba-binary\envs\conda-lock-env\envs
                          C:\Users\**username**\.conda\envs
                          C:\Users\**username**\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/24.5.0 requests/2.32.3 CPython/3.12.4 Windows/10 Windows/10.0.19045 solver/libmamba conda-libmamba-solver/24.1.0 libmambapy/1.5.8
          administrator : False
             netrc file : None
           offline mode : False

Conda Config

==> **redacted**\micromamba-binary\envs\conda-lock-env\.condarc <==
ssl_verify: False
report_errors: False

Conda list

# packages in environment at **redacted**\micromamba-binary\envs\conda-lock-env:
#
# Name                    Version                   Build  Channel
annotated-types           0.7.0                    pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
archspec                  0.2.3                    pypi_0    pypi
backports                 1.0                pyhd8ed1ab_3    https://**internal_conda_repo**
backports-tarfile         1.0.0                    pypi_0    pypi
backports.tarfile         1.0.0              pyhd8ed1ab_1    https://**internal_conda_repo**
boltons                   24.0.0                   pypi_0    pypi
brotli                    1.1.0                    pypi_0    pypi
brotli-python             1.1.0           py312h53d5487_1    https://**internal_conda_repo**
bzip2                     1.0.8                hcfcfb64_5    https://**internal_conda_repo**
ca-certificates           2024.6.2             h56e8100_0    https://**internal_conda_repo**
cachecontrol              0.14.0                   pypi_0    pypi
cachecontrol-with-filecache 0.14.0             pyhd8ed1ab_1    https://**internal_conda_repo**
cachy                     0.3.0                    pypi_0    pypi
certifi                   2024.6.2                 pypi_0    pypi
cffi                      1.16.0                   pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
click-default-group       1.2.4                    pypi_0    pypi
clikit                    0.6.2                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
conda                     24.5.0                   pypi_0    pypi
conda-libmamba-solver     24.1.0                   pypi_0    pypi
conda-lock                2.5.7                    pypi_0    pypi
conda-package-handling    2.3.0                    pypi_0    pypi
conda-package-streaming   0.10.0                   pypi_0    pypi
crashtest                 0.4.1                    pypi_0    pypi
distlib                   0.3.8                    pypi_0    pypi
distro                    1.9.0                    pypi_0    pypi
ensureconda               1.4.4                    pypi_0    pypi
filelock                  3.15.4                   pypi_0    pypi
fmt                       10.2.1               h181d51b_0    https://**internal_conda_repo**
frozendict                2.4.4                    pypi_0    pypi
gitdb                     4.0.11                   pypi_0    pypi
gitpython                 3.1.43                   pypi_0    pypi
html5lib                  1.1                      pypi_0    pypi
idna                      3.7                      pypi_0    pypi
importlib-metadata        7.2.1                    pypi_0    pypi
importlib-resources       6.4.0                    pypi_0    pypi
importlib_metadata        7.2.1                hd8ed1ab_0    https://**internal_conda_repo**
importlib_resources       6.4.0              pyhd8ed1ab_0    https://**internal_conda_repo**
jaraco-classes            3.4.0                    pypi_0    pypi
jaraco-context            5.3.0                    pypi_0    pypi
jaraco-functools          4.0.0                    pypi_0    pypi
jaraco.classes            3.4.0              pyhd8ed1ab_1    https://**internal_conda_repo**
jaraco.context            5.3.0              pyhd8ed1ab_1    https://**internal_conda_repo**
jaraco.functools          4.0.0              pyhd8ed1ab_0    https://**internal_conda_repo**
jinja2                    3.1.4                    pypi_0    pypi
jsonpatch                 1.33                     pypi_0    pypi
jsonpointer               3.0.0                    pypi_0    pypi
keyring                   25.2.1                   pypi_0    pypi
krb5                      1.21.2               heb0366b_0    https://**internal_conda_repo**
libarchive                3.7.4                haf234dc_0    https://**internal_conda_repo**
libcurl                   8.8.0                hd5e4a3a_0    https://**internal_conda_repo**
libexpat                  2.6.2                h63175ca_0    https://**internal_conda_repo**
libffi                    3.4.2                h8ffe710_5    https://**internal_conda_repo**
libiconv                  1.17                 hcfcfb64_2    https://**internal_conda_repo**
libmamba                  1.5.8                h3f09ed1_0    https://**internal_conda_repo**
libmambapy                1.5.8                    pypi_0    pypi
libsolv                   0.7.29               h0ea2cb4_0    https://**internal_conda_repo**
libsqlite                 3.46.0               h2466b09_0    https://**internal_conda_repo**
libssh2                   1.11.0               h7dfc565_0    https://**internal_conda_repo**
libxml2                   2.12.7               h283a6d9_1    https://**internal_conda_repo**
libzlib                   1.3.1                h2466b09_1    https://**internal_conda_repo**
lz4-c                     1.9.4                hcfcfb64_0    https://**internal_conda_repo**
lzo                       2.10              hcfcfb64_1001    https://**internal_conda_repo**
markupsafe                2.1.5                    pypi_0    pypi
menuinst                  2.1.1                    pypi_0    pypi
more-itertools            10.3.0                   pypi_0    pypi
msgpack                   1.0.8                    pypi_0    pypi
msgpack-python            1.0.8           py312hd5eb7cc_0    https://**internal_conda_repo**
openssl                   3.3.1                h2466b09_0    https://**internal_conda_repo**
packaging                 24.1                     pypi_0    pypi
pastel                    0.2.1                    pypi_0    pypi
pip                       24.0                     pypi_0    pypi
pkginfo                   1.11.1                   pypi_0    pypi
platformdirs              4.2.2                    pypi_0    pypi
pluggy                    1.5.0                    pypi_0    pypi
pybind11-abi              4                    hd8ed1ab_3    https://**internal_conda_repo**
pycosat                   0.6.6                    pypi_0    pypi
pycparser                 2.22                     pypi_0    pypi
pydantic                  2.7.4                    pypi_0    pypi
pydantic-core             2.18.4                   pypi_0    pypi
pylev                     1.4.0                    pypi_0    pypi
pysocks                   1.7.1                    pypi_0    pypi
python                    3.12.4          h889d299_0_cpython    https://**internal_conda_repo**
python_abi                3.12                    4_cp312    https://**internal_conda_repo**
pywin32-ctypes            0.2.2                    pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
reproc                    14.2.4.post0         hcfcfb64_1    https://**internal_conda_repo**
reproc-cpp                14.2.4.post0         h63175ca_1    https://**internal_conda_repo**
requests                  2.32.3                   pypi_0    pypi
ruamel-yaml               0.18.6                   pypi_0    pypi
ruamel-yaml-clib          0.2.8                    pypi_0    pypi
ruamel.yaml               0.18.6          py312he70551f_0    https://**internal_conda_repo**
ruamel.yaml.clib          0.2.8           py312he70551f_0    https://**internal_conda_repo**
setuptools                70.1.0                   pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
smmap                     5.0.0                    pypi_0    pypi
tk                        8.6.13               h5226925_1    https://**internal_conda_repo**
tomli                     2.0.1                    pypi_0    pypi
tomlkit                   0.12.5                   pypi_0    pypi
toolz                     0.12.1                   pypi_0    pypi
tqdm                      4.66.4                   pypi_0    pypi
truststore                0.8.0                    pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
typing_extensions         4.12.2             pyha770c72_0    https://**internal_conda_repo**
tzdata                    2024a                h0c530f3_0    https://**internal_conda_repo**
ucrt                      10.0.22621.0         h57928b3_0    https://**internal_conda_repo**
urllib3                   1.26.19                  pypi_0    pypi
vc                        14.3                h8a93ad2_20    https://**internal_conda_repo**
vc14_runtime              14.40.33810         ha82c5b3_20    https://**internal_conda_repo**
virtualenv                20.26.3                  pypi_0    pypi
vs2015_runtime            14.40.33810         h3bf8584_20    https://**internal_conda_repo**
webencodings              0.5.1                    pypi_0    pypi
wheel                     0.43.0                   pypi_0    pypi
win-inet-pton             1.1.0                    pypi_0    pypi
win_inet_pton             1.1.0              pyhd8ed1ab_6    https://**internal_conda_repo**
xz                        5.2.6                h8d14728_0    https://**internal_conda_repo**
yaml                      0.2.5                h8ffe710_2    https://**internal_conda_repo**
yaml-cpp                  0.8.0                h63175ca_0    https://**internal_conda_repo**
zipp                      3.19.2                   pypi_0    pypi
zstandard                 0.22.0                   pypi_0    pypi
zstd                      1.5.6                h0ea2cb4_0    https://**internal_conda_repo**

Additional Context

Everyhing is running within a micromamba generated env.

diego-comlet commented 5 months ago

I think the issue could come from the differning ways Conda manages the SSL, and Poetry does internally

diego-comlet commented 5 months ago

The following env variables, have been tried to no avail

REQUESTS_CA_BUNDLE=${CERT} 
SSL_CERT_FILE=${CERT} 
CURL_CA_BUNDLE=${CERT} 
PIP_CERT=${CERT}

And also in a separate test

SSL_NO_VERIFY=1

3ll3d00d commented 2 months ago

@diego-comlet looks like CONDA_FLAGS="--insecure" works as these get appended to the generated command line according to

https://github.com/conda/conda-lock/blob/29b78b198911f4976541bb8f8bb3e4d265f02f8f/conda_lock/invoke_conda.py#L189