pip integration: conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command ... returned non-zero exit status 128

mjlbach commented 10 months ago

Checklist

[X] I added a descriptive title
[X] I searched open reports and couldn't find a duplicate

What happened?

If you try to conda-lock the following environment:

environment.yml

channels:
  - defaults
  - pytorch
  - nvidia
  - conda-forge
dependencies:
  - python=3.10
  - pip
  - pytorch=2.0.1=py3.10_cuda11.8_cudnn8.7.0_0
  - torchvision=0.15.2=py310_cu118
  - pytorch-cuda=11.8
  - pip:
    - git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

conda-lock --mamba -p linux-64

You will receive the following error:

❯ conda-lock --mamba -p linux-64
/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/pydantic/_internal/_config.py:269: UserWarning: Valid config keys have changed in V2:
* 'allow_mutation' has been removed
  warnings.warn(message, UserWarning)
Locking dependencies for ['linux-64']...
INFO:conda_lock.conda_solver:linux-64 using specs ['python 3.10.*', 'pip *', 'pytorch 2.0.1 py3.10_cuda11.8_cudnn8.7.0_0', 'torchvision 0.15.2 py310_cu118', 'pytorch-cuda 11.8.*']
Traceback (most recent call last):
  File "/home/michael/.local/bin/conda-lock", line 8, in <module>
    sys.exit(main())
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1332, in lock
    lock_func(
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1040, in run_lock
    make_lock_files(
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 381, in make_lock_files
    lock_content = lock_content | create_lockfile_from_spec(
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 787, in create_lockfile_from_spec
    deps = _solve_for_arch(
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 721, in _solve_for_arch
    pip_deps = solve_pypi(
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/pypi_solver.py", line 335, in solve_pypi
    result = s.solve(use_latest=to_update)
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/solver.py", line 65, in solve
    packages, depths = self._solve(use_latest=use_latest)
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/solver.py", line 233, in _solve
    result = resolve_version(
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/__init__.py", line 7, in resolve_version
    return solver.solve()
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 84, in solve
    next = self._choose_package_version()
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/mixology/version_solver.py", line 400, in _choose_package_version
    version = self._provider.complete_package(version)
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/provider.py", line 452, in complete_package
    self.search_for_vcs(r)
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/provider.py", line 163, in search_for_vcs
    package = self.get_package_from_vcs(
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/puzzle/provider.py", line 193, in get_package_from_vcs
    git.clone(url, tmp_dir)
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/core/vcs/git.py", line 262, in clone
    return self.run("clone", "--recurse-submodules", "--", repository, str(dest))
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/core/vcs/git.py", line 356, in run
    subprocess.check_output(
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/utils/_compat.py", line 217, in run
    raise CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['git', 'clone', '--recurse-submodules', '--', 'https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch', '/tmp/pypoetry-git-torchc39fhmwj']' returned non-zero exit status 128.

Conda Info

mamba 1.4.2
conda 23.3.1

Conda Config

❯ conda config --show-sources
==> /home/michael/mambaforge/.condarc <==
channels:
  - conda-forge

Conda list

❯ conda list --show-channel-urls
# packages in environment at /home/michael/mambaforge:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
boltons                   23.0.0             pyhd8ed1ab_0    conda-forge
brotli-python             1.0.9           py310hd8f1fbe_9    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_3    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     23.3.1          py310hff52083_0    conda-forge
conda-libmamba-solver     23.3.0             pyhd8ed1ab_0    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
cryptography              41.0.3          py310h75e40e8_0    conda-forge
fmt                       9.1.0                h924138e_0    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
jsonpointer               2.0                        py_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libarchive                3.6.2                h039dbb9_1    conda-forge
libcurl                   8.2.1                hca28451_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libmamba                  1.4.2                hcea66bb_0    conda-forge
libmambapy                1.4.2           py310h1428755_0    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsolv                   0.7.24               hfc55251_1    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxml2                   2.11.5               h0d562d8_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     1.4.2           py310h51d5547_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
openssl                   3.1.2                hd590300_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pluggy                    1.2.0              pyhd8ed1ab_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.4           py310h5764c6d_1    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.12         hd12c33a_0_cpython    conda-forge
python_abi                3.10                    3_cp310    conda-forge
readline                  8.2                  h8228510_1    conda-forge
reproc                    14.2.4               h0b41bf4_0    conda-forge
reproc-cpp                14.2.4               hcb278e6_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.17.32         py310h2372a71_0    conda-forge
ruamel.yaml.clib          0.2.7           py310h1fa729e_1    conda-forge
setuptools                68.1.2             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   2.0.4              pyhd8ed1ab_0    conda-forge
wheel                     0.41.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml-cpp                  0.7.0                h27087fc_2    conda-forge
zstandard                 0.19.0          py310h1275a96_2    conda-forge
zstd                      1.5.2                hfc55251_7    conda-forge



### Additional Context

_No response_

mjlbach commented 10 months ago

Also happens with other git repos, e.g. git+https://github.com/luca-medeiros/lang-segment-anything.git

Of note, this and https://github.com/conda/conda-lock/issues/395 are the only blockers for using conda-lock for my project.

anuraaga commented 6 months ago

@maresb This is an old issue so hoping to bring it up, I ran into the same issue. It seems that when parsing environment.yml, the # is not handled correctly and gets treated as part of the repository name.

maresb commented 6 months ago

Thanks @anuraaga for the info, and I'm sorry you're having problems with conda-lock.

It seems that when parsing environment.yml, the # is not handled correctly and gets treated as part of the repository name.

Would you be able to explain in more detail exactly what you mean by this comment? Thanks!

anuraaga commented 6 months ago

Ah sorry it wasn't clear. PIP supports parameters for vcs dependencies past #

https://pip.pypa.io/en/stable/cli/pip_install/#vcs-support

For example

python -m pip install -e 'git+https://git.repo/some_repo.git#egg=subdir&subdirectory=subdir_path' # install a python package from a repo subdirectory

But we are from the error message that it seems git clone is being executed with the contents of # which should not be part of the clone command, the final line of the original post

File "/home/michael/.local/pipx/venvs/conda-lock/lib/python3.10/site-packages/conda_lock/_vendor/poetry/utils/_compat.py", line 217, in run
    raise CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['git', 'clone', '--recurse-submodules', '--', 'https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch', '/tmp/pypoetry-git-torchc39fhmwj']' returned non-zero exit status 128

I can confirm having similar behavior when trying to use #subdirectory with a similar dependency

maresb commented 6 months ago

Thanks a lot @anuraaga for the detailed explanation!

The error seems to be occurring within the old version of Poetry that I vendored into conda-lock. I suspect that upgrading this vendored Poetry would resolve the issue. I have been meaning to tackle this for some time now. I hope to be able to finally get to it in early 2024, but there are no promises.

conda / conda-lock