conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command returned non-zero exit status 1

sebastian-luna-valero commented 1 year ago

Checklist

[X] I added a descriptive title
[X] I searched open reports and couldn't find a duplicate

What happened?

Hello,

I cannot lock the following environment file:

name: test
channels:
  - conda-forge
dependencies:
  - rasterio

with:

conda-lock --kind explicit --platform linux-64 --file env.yml

Output is attached: out.txt

However, mamba correctly resolves the environment:

mamba env create -n test01 -f env3.yml

It works when I replace --platform linux-64 with:

--platform osx-64
--platform win-64
--platform osx-arm64

Could you please help?

Thanks, Sebastian

Conda Info

active environment : conda-lock
    active env location : /tmp/tmp.kRoJ2PVs1C/conda-install/envs/conda-lock
            shell level : 1
       user config file : /home/sebastian/.condarc
 populated config files : 
          conda version : 23.1.0
    conda-build version : not installed
         python version : 3.10.10.final.0
       virtual packages : __archspec=1=x86_64
                          __glibc=2.31=0
                          __linux=5.14.0=0
                          __unix=0=0
       base environment : /tmp/tmp.kRoJ2PVs1C/conda-install  (writable)
      conda av data dir : /tmp/tmp.kRoJ2PVs1C/conda-install/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /tmp/tmp.kRoJ2PVs1C/conda-install/pkgs
                          /home/sebastian/.conda/pkgs
       envs directories : /tmp/tmp.kRoJ2PVs1C/conda-install/envs
                          /home/sebastian/.conda/envs
               platform : linux-64
             user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.10 Linux/5.14.0-1058-oem ubuntu/20.04.2 glibc/2.31
                UID:GID : 1001:1001
             netrc file : None
           offline mode : False

Conda Config

No response

Conda list

conda list --show-channel-urls
# packages in environment at /tmp/tmp.kRoJ2PVs1C/conda-install/envs/conda-lock:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             5.1                       1_gnu    defaults
appdirs                   1.4.4              pyhd3eb1b0_0    defaults
brotlipy                  0.7.0           py311h5eee18b_1002    defaults
bzip2                     1.0.8                h7b6447c_0    defaults
ca-certificates           2023.01.10           h06a4308_0    defaults
cachecontrol              0.12.11         py311h06a4308_1    defaults
cachecontrol-with-filecache 0.12.11         py311h06a4308_1    defaults
cachy                     0.3.0              pyhd3eb1b0_0    defaults
certifi                   2022.12.7       py311h06a4308_0    defaults
cffi                      1.15.1          py311h5eee18b_3    defaults
charset-normalizer        2.0.4              pyhd3eb1b0_0    defaults
click                     8.0.4           py311h06a4308_0    defaults
click-default-group       1.2.2           py311h06a4308_0    defaults
clikit                    0.6.2                      py_0    defaults
conda-lock                1.4.0           py311h06a4308_0    defaults
crashtest                 0.3.1              pyhd3eb1b0_1    defaults
cryptography              39.0.1          py311h9ce1e76_0    defaults
dbus                      1.13.18              hb2f20db_0    defaults
distlib                   0.3.6           py311h06a4308_0    defaults
ensureconda               1.4.3           py311h06a4308_0    defaults
expat                     2.4.9                h6a678d5_0    defaults
filelock                  3.9.0           py311h06a4308_0    defaults
flit-core                 3.8.0           py311h06a4308_0    defaults
glib                      2.69.1               he621ea3_2    defaults
html5lib                  1.1                pyhd3eb1b0_0    defaults
idna                      3.4             py311h06a4308_0    defaults
importlib-metadata        6.0.0           py311h06a4308_0    defaults
importlib_metadata        6.0.0                hd3eb1b0_0    defaults
jaraco.classes            3.2.1              pyhd3eb1b0_0    defaults
jeepney                   0.7.1              pyhd3eb1b0_0    defaults
jinja2                    3.1.2           py311h06a4308_0    defaults
keyring                   23.13.1         py311h06a4308_0    defaults
ld_impl_linux-64          2.38                 h1181459_1    defaults
libffi                    3.4.2                h6a678d5_6    defaults
libgcc-ng                 11.2.0               h1234567_1    defaults
libgomp                   11.2.0               h1234567_1    defaults
libstdcxx-ng              11.2.0               h1234567_1    defaults
libuuid                   1.41.5               h5eee18b_0    defaults
lockfile                  0.12.2          py311h06a4308_0    defaults
markupsafe                2.1.1           py311h5eee18b_0    defaults
more-itertools            8.12.0             pyhd3eb1b0_0    defaults
msgpack-python            1.0.3           py311hdb19cb5_0    defaults
ncurses                   6.4                  h6a678d5_0    defaults
openssl                   1.1.1t               h7f8727e_0    defaults
packaging                 23.0            py311h06a4308_0    defaults
pastel                    0.2.1                      py_0    defaults
pcre                      8.45                 h295c915_0    defaults
pip                       23.0.1          py311h06a4308_0    defaults
pkginfo                   1.9.6           py311h06a4308_0    defaults
platformdirs              2.5.2           py311h06a4308_0    defaults
pycparser                 2.21               pyhd3eb1b0_0    defaults
pydantic                  1.10.2          py311h5eee18b_0    defaults
pylev                     1.3.0                      py_0    defaults
pyopenssl                 23.0.0          py311h06a4308_0    defaults
pysocks                   1.7.1           py311h06a4308_0    defaults
python                    3.11.0               h7a1cb2a_3    defaults
pyyaml                    6.0             py311h5eee18b_1    defaults
readline                  8.2                  h5eee18b_0    defaults
requests                  2.28.1          py311h06a4308_1    defaults
ruamel.yaml               0.17.21         py311h5eee18b_0    defaults
secretstorage             3.3.1           py311h06a4308_1    defaults
setuptools                65.6.3          py311h06a4308_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
sqlite                    3.41.1               h5eee18b_0    defaults
tk                        8.6.12               h1ccaba5_0    defaults
tomlkit                   0.11.1          py311h06a4308_0    defaults
toolz                     0.12.0          py311h06a4308_0    defaults
typing-extensions         4.4.0           py311h06a4308_0    defaults
typing_extensions         4.4.0           py311h06a4308_0    defaults
tzdata                    2022g                h04d1e81_0    defaults
urllib3                   1.26.14         py311h06a4308_0    defaults
virtualenv                20.17.1         py311h06a4308_0    defaults
webencodings              0.5.1           py311h06a4308_1    defaults
wheel                     0.38.4          py311h06a4308_0    defaults
xz                        5.2.10               h5eee18b_1    defaults
yaml                      0.2.5                h7b6447c_0    defaults
zipp                      3.11.0          py311h06a4308_0    defaults
zlib                      1.2.13               h5eee18b_0    defaults

Additional Context

No response

maresb commented 1 year ago

This looks like a possible duplicate of #381. Could you please try downgrading Conda to 22.9 and see if that fixes it?

In any case, this is a nice, short example.

epassaro commented 1 year ago

I have the same problem with a completely different environment. I don't know why, because conda-lock was working fine in the last weeks, and I didn't update my Miniforge installation.

maresb commented 1 year ago

@epassaro, what is your Conda version?

epassaro commented 1 year ago

Tried with 22.9 but didn't work.

maresb commented 1 year ago

That's interesting. Could you share your environment file, your platform, and the versions of conda-lock, mamba, and conda? It'd be great to be able to reproduce this.

sebastian-luna-valero commented 1 year ago

Hi,

The steps below worked for me:

cd working-dir
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -p conda-install
source conda-install/etc/profile.d/conda.sh 
conda install conda=22.9 mamba -c conda-forge 
mamba create -n conda-lock conda-lock -c conda-forge
conda activate conda-lock
conda-lock --kind explicit --platform linux-64 --file env.yml

Best regards, Sebastian

maresb commented 1 year ago

Hi @sebastian-luna-valero, could you please clarify what you mean by "worked"? Is this intended to reproduce @epassaro's failure, or to produce a lockfile. (I tried with your instructions and the lock succeeded.)

sebastian-luna-valero commented 1 year ago

Hi @maresb

I mean I was able to create a lockfile, which was my initial issue.

Best regards, Sebastian

maresb commented 1 year ago

Great, thanks for clarifying!

In that case I'd still like to know what's happening with @epassaro's situation where the 22.9 workaround didn't work.

epassaro commented 1 year ago

Hi again @maresb !

OS: openSUSE Tumbleweed, latest snapshot
I installed the latest Mambaforge in /opt/mambaforge
Created a new environment called lock with the command: mamba create -n lock python=3.9 conda=22.9 conda-lock=1.4
Activated the environment and tried to lock this environment.yml file with: conda-lock lock -f environment.yml -p linux-64

But didn't work, same result.

maresb commented 1 year ago

Thanks a lot @epassaro for the additional info.

I suspect what's happening is that conda-lock is discovering the conda which comes bundled with conda-forge rather than the conda from your lock environment.

Could you please try downgrading the conda from your base environment in /opt/mambaforge and see if that resolves this problem?

epassaro commented 1 year ago

Tried with Mambaforge 22.9.0-3 (Dec 2022) and got a different error:

╰─ conda-lock lock -f environment.yml -p linux-64
Locking dependencies for ['linux-64']...
INFO:conda_lock.conda_solver:linux-64 using specs ['python 3.9.*', 'setuptools', 'pip *', 'gdown >=4.7.1', 'tesseract >=5.3.0', 'pytesseract >=0.3.10', 'numpy >=1.21.6', 'scipy >=1.4.1', 'pandas >=1.1.4', 'matplotlib-base >=3.2.2', 'seaborn >=0.11.0', 'py-opencv >=4.6.0', 'pytorch >=1.7.0', 'torchvision >=0.8.1', 'pillow >=7.1.2', 'requests >=2.23.0', 'pyyaml >=5.3.1', 'tqdm >=4.64.0', 'notebook >=6.5.3', 'sentry-sdk >=1.17.0', 'psutil >=5.9.4']
Traceback (most recent call last):
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/conda_lock/src_parser/__init__.py", line 488, in seperator_munge_get
    return d[key]
KeyError: 'pytorch'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/conda_lock/src_parser/__init__.py", line 491, in seperator_munge_get
    return d[key.replace("-", "_")]
KeyError: 'pytorch'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/epassaro/.conda/envs/lock/bin/conda-lock", line 10, in <module>
    sys.exit(main())
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1353, in lock
    lock_func(
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1083, in run_lock
    make_lock_files(
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 408, in make_lock_files
    lock_content = lock_content | create_lockfile_from_spec(
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 801, in create_lockfile_from_spec
    deps = _solve_for_arch(
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 737, in _solve_for_arch
    pip_deps = solve_pypi(
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/conda_lock/pypi_solver.py", line 327, in solve_pypi
    src_parser._apply_categories(requested=pip_specs, planned=planned)
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/conda_lock/src_parser/__init__.py", line 502, in _apply_categories
    for dep in seperator_munge_get(planned, item).dependencies
  File "/home/epassaro/.conda/envs/lock/lib/python3.10/site-packages/conda_lock/src_parser/__init__.py", line 493, in seperator_munge_get
    return d[key.replace("_", "-")]
KeyError: 'pytorch'

maresb commented 1 year ago

Yes, that's progress.

Could you please try with the branch from #253?

pip install git+https://github.com/romain-intel/conda-lock@fix/pypi-conda-pkg-confusion

epassaro commented 1 year ago

It worked! Thanks <3

I have some -I think- unrelated questions: is the [pip_support] extra still a thing? How should be used? In which conda-lock version was introduced?

maresb commented 1 year ago

Yay!!! Glad it's working for you now. Hopefully we can get all these issues solved soon so that jumping through all these hoops is no longer necessary.

Regarding pip_support, it's no longer a thing since we vendored (inserted the source code from) Poetry. The pip_support extra just caused Poetry to be pip installed alongside of conda-lock but now with vendoring it no longer serves any purpose.

epassaro commented 1 year ago

I noticed that installing the environment doing mamba create -n <environment-name> --file conda-linux-64.lock does not install pip packages, however these packages appeared commented at the bottom of the file.

I need to pass an extra flag or something to install everything with a single command?

EDIT: forgot to mention i did conda-lock render conda-lock.yml before.

maresb commented 1 year ago

To install from a lockfile with pip packages, you should use either conda-lock install or micromamba.

epassaro commented 1 year ago

Thanks!

valeriupredoi commented 1 year ago

hey guys, heads up - since I have been running into the same issue (the poetry-related one, in the OP): I confirm that reverting to conda=22.9, conda-lock=1.4 runs fine, also, more importantly - updating to the latest conda=23.3.1 conda-lock=1.4 works very well too, it seems there is a certain issue with conda=23.1 :+1:

maresb commented 1 year ago

Thank you very much for the confirmation, @valeriupredoi, it is much appreciated!

Perhaps we should be checking the Conda version and raise a warning in case the affected version is installed.

valeriupredoi commented 1 year ago

@maresb my pleasure, mate! That sounds like a good idea to me! Maybe even raise a ticket with conda too, if you have an idea what may be going wrong (although, knowing their 3k+ issues, I doubt there may be any timely action taken :grin: )

conda / conda-lock