maresb
commented
1 year ago I'm not able to reproduce this, it's successfully locking for me. Could you please try deleting your conda-lock.yml file and trying again?
Closed ivanrhsosa closed 1 year ago
maresb
commented
1 year ago I'm not able to reproduce this, it's successfully locking for me. Could you please try deleting your conda-lock.yml file and trying again?
maresb
commented
1 year ago Or even better, rename it so that in case it's corrupt we can debug how it became corrupt.
srilman
commented
1 year ago I was able to reproduce this error on an M1 using the latest version of conda-lock. I found a slightly more minimal reproducer:
channels:
- conda-forge
dependencies:
- python 3.10.*
- grpcioNote that grpcio 1.51.* is a subdependency of tensorflow. The full standard error output is the following (I've redacted some paths and skipped some parts):
Locking dependencies for ['osx-64']...
INFO:conda_lock.conda_solver:osx-64 using specs ['python 3.10.*', 'grpcio']
Failed to parse json, Extra data: line 145 column 1 (char 10582)
Could not lock the environment for platform osx-64
{
"command": ".../mambaforge/condabin/mamba create --prefix ... python 3.10.* grpcio",
"conda_info": {...},
"error": ...,
"exception_name": "KeyError",
"exception_type": "<class 'KeyError'>",
"traceback": ...
}
{
"actions": {
"FETCH": [...]
"LINK": [...],
"success": true
}
STDERR:
PackageRecord(_hash=-3434338967460783354, name='__osx', version='0', build='0', build_number=0, channel=Channel("@"), subdir='osx-64', fn='__osx', md5='12345678901234567890123456789012', package_type='virtual_system')
Traceback (most recent call last):
File ".../.local/pipx/.cache/e9784d76eb264dc/bin/conda-lock", line 8, in <module>
sys.exit(main())
...
File ".../.local/pipx/.cache/e9784d76eb264dc/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1353, in lock
lock_func(
File ".../.local/pipx/.cache/e9784d76eb264dc/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 1083, in run_lock
make_lock_files(
File ".../.local/pipx/.cache/e9784d76eb264dc/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 408, in make_lock_files
lock_content = lock_content | create_lockfile_from_spec(
File ".../.local/pipx/.cache/e9784d76eb264dc/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 801, in create_lockfile_from_spec
deps = _solve_for_arch(
File ".../.local/pipx/.cache/e9784d76eb264dc/lib/python3.10/site-packages/conda_lock/conda_lock.py", line 723, in _solve_for_arch
conda_deps = solve_conda(
File ".../.local/pipx/.cache/e9784d76eb264dc/lib/python3.10/site-packages/conda_lock/conda_solver.py", line 168, in solve_conda
dry_run_install = solve_specs_for_arch(
File ".../.local/pipx/.cache/e9784d76eb264dc/lib/python3.10/site-packages/conda_lock/conda_solver.py", line 355, in solve_specs_for_arch
proc.check_returncode()
File ".../.local/pipx/.cache/e9784d76eb264dc/lib/python3.10/site-packages/conda_lock/_vendor/poetry/utils/_compat.py", line 168, in check_returncode
raise CalledProcessError(
conda_lock._vendor.poetry.utils._compat.CalledProcessError: Command '['.../mambaforge/condabin/mamba', 'create', '--prefix', '/var/folders/by/xw3fz9wx0sz3nj6mvd941gpm0000gn/T/tmpv9kd1tj_/prefix', '--dry-run', '--json', '--override-channels', '--channel', 'conda-forge', '--channel', 'file:///var/folders/by/xw3fz9wx0sz3nj6mvd941gpm0000gn/T/tmpffsnmedz', 'python 3.10.*', 'grpcio']' returned non-zero exit status 1.I see from the request JSON output that this is the specification for grpcio:
"build": "py310h7a76584_1",
"build_number": 1,
"build_string": "py310h7a76584_1",
"channel": "https://conda.anaconda.org/conda-forge/osx-64",
"constrains": null,
"depends": [
"__osx >=10.9",
"libcxx >=14.0.6",
"libgrpc 1.52.1 h5bc3d57_1",
"python >=3.10,<3.11.0a0",
"python_abi 3.10.* *_cp310"
],
"fn": "grpcio-1.52.1-py310h7a76584_1.conda",
"license": "Apache-2.0",
"md5": "2624342586b1bc2766bd6d26a4213d36",
"name": "grpcio",
"sha256": "6d783d79fbe7656099031204aae73735e4a4e466794d9fc8019a31e82c94331b",
"size": 700661,
"subdir": "osx-64",
"timestamp": 1677501826,
"track_features": "",
"url": "https://conda.anaconda.org/conda-forge/osx-64/grpcio-1.52.1-py310h7a76584_1.conda",
"version": "1.52.1"Looking at the error message, I wonder if the problem is the "___osx >= 10.9" dependency on grpcio. I am running on macOS Ventura 13.2.1 if that is relavent.
srilman
commented
1 year ago Found mamba's traceback in the skipped parts:
Traceback (most recent call last):
File ".../mambaforge/lib/python3.10/site-packages/conda/_vendor/boltons/setutils.py", line 247, in remove
didx = self.item_index_map.pop(item)
KeyError: PackageRecord(_hash=-3434338967460783354, name='__osx', version='0', build='0', build_number=0, channel=Channel("@"), subdir='osx-64', fn='__osx', md5='12345678901234567890123456789012', package_type='virtual_system')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ".../mambaforge/lib/python3.10/site-packages/conda/exceptions.py", line 1118, in __call__
return func(*args, **kwargs)
File ".../mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 936, in exception_converter
raise e
File ".../mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 929, in exception_converter
exit_code = _wrapped_main(*args, **kwargs)
File ".../mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 887, in _wrapped_main
result = do_call(parsed_args, p)
File ".../mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 754, in do_call
exit_code = create(args, parser)
File ".../mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 630, in create
return install(args, parser, "create")
File ".../mambaforge/lib/python3.10/site-packages/mamba/mamba.py", line 588, in install
conda_transaction = to_txn(
File ".../mambaforge/lib/python3.10/site-packages/mamba/utils.py", line 436, in to_txn
final_precs = compute_final_precs(
File ".../mambaforge/lib/python3.10/site-packages/mamba/utils.py", line 371, in compute_final_precs
final_precs.remove(i_rec)
File ".../mambaforge/lib/python3.10/site-packages/conda/_vendor/boltons/setutils.py", line 249, in remove
raise KeyError(item)
KeyError: PackageRecord(_hash=-3434338967460783354, name='__osx', version='0', build='0', build_number=0, channel=Channel("@"), subdir='osx-64', fn='__osx', md5='12345678901234567890123456789012', package_type='virtual_system')
croth1
commented
1 year ago @srilman's error reminds me of https://github.com/conda/conda-lock/issues/292, which was caused by a regression in conda when moving virtual packages to their new plugin system. A strong indication for that would be if it works with conda=22.9, but breaks with conda>=22.11.
croth1
commented
1 year ago roping in @dholth - I think this is conceptually the same error you fixed for libc back in january: https://github.com/conda/conda/pull/12293.
In the pre-plugin version, CONDA_OVERRIDE_OSX="" would not export a virtual package, but the new version does:
old: https://github.com/conda/conda/blob/60e11b4f1859874c1cb1a3ae324e599e0867a086/conda/core/index.py#L181-L184 new: https://github.com/conda/conda/blob/main/conda/plugins/virtual_packages/osx.py#L11-L18
dholth
commented
1 year ago Well, I fixed it in 23.1; can conda-lock's vendored conda be updated?
Hopefully we don't have the wrong packages saved to lock files?
srilman
commented
1 year ago @maresb Are you familiar with how Conda was vendored? I see that we only use the vendored Conda for the MatchSpec class and the toposort function
srilman
commented
1 year ago Locally, I upgraded conda to 2023.1 and mamba to 1.3.1. Using mamba, I still see the same error. However, this time I used conda and it successfully locked, so @ivanrhsosa consider doing that. You can force it to use conda by passing in the --no-mamba flag.
maresb
commented
1 year ago I'm somewhat familiar with the Conda vendoring process. @mariusvniekerk vendored Conda and I vendored Poetry. I was surprised that we use so little of the vendored Conda. Toposort should in principle be easy to replace. For MatchSpec I'm not sure how deeply that goes.
But it's easy to get confused... are we relying on the external Conda pacakge or the vendored one?
srilman
commented
1 year ago I think? this is most likely a problem with external's Conda, since the error comes from the subprocess where we call mamba create or conda create. However, it is frustrating that using external Mamba is still failing.
Toposort looks very easy to replace. MatchSpec seems much harder; it imports many things from the core set of modules like common, base, exceptions, auxlib and so on.
ivanrhsosa
commented
1 year ago --no-mamba
using the --no-mamba flag worked and successfully locked osx-64. Thanks @srilman
leroyvn
commented
1 year ago It worked for me as well.
maresb
commented
1 year ago It's great that you found a workaround! However, let's keep this open until we find a proper solution. It would be nice to understand why Mamba is failing and solve the underlying problem.
leeleavitt
commented
1 year ago Also turning off mamba support locked my dependencies, much slower though. Would love mamba.
leeleavitt
commented
1 year ago Is there anything i can provide to help debug this issue?
maresb
commented
1 year ago For me, installing conda=22.9 seems to be a good workaround.
I haven't had a chance to look into this, but I'd be very interested to know why it doesn't work with Conda 23.1 despite the fix on the Conda side. Do we also need to adjust something on our side?
lesteve
commented
1 year ago As noted by @croth1 in https://github.com/conda/conda-lock/issues/381#issuecomment-1454564219 it feels like the osx issue is very similar to the one that was fixed by @dholth for glibc in https://github.com/conda/conda/pull/12293.
I have opened https://github.com/conda/conda/pull/12715 to hopefully fix the osx one.
maresb
commented
1 year ago This should be fixed now by installing the latest Conda. Ref: https://github.com/sagemath/sage/pull/35986#issuecomment-1656713210
Checklist
What happened?
I cannot lock the environment for platform osx-64 but I can lock successfully the environment for platforms linux-64 and osx-arm64
command and error message $ conda-lock -f environment.yml -p osx-64
Locking dependencies for ['osx-64']... INFO:conda_lock.conda_solver:osx-64 using specs ['python 3.10.', 'mamba', 'pip', 'conda-lock', 'py-xgboost-cpu', 'tensorflow 2.10.'] Failed to parse json, Extra data: line 100 column 1 (char 6597) Could not lock the environment for platform osx-64 { "command": "/usr/local/Caskroom/miniforge/base/bin/mamba create --prefix /var/folders/91/x9jf55z17vg9l94bpd_c1qp40000gp/T/tmpz20z5pw4/prefix --dry-run --json --override-channels --channel conda-forge --channel nodefaults --channel file:///var/folders/91/x9jf55z17vg9l94bpd_c1qp40000gp/T/tmpdb9saiyz python 3.10. mamba pip conda-lock py-xgboost-cpu tensorflow 2.10.",
environment.yml
Conda Info
Conda Config
Conda list
Additional Context
No response