Fails to resolve for dependency with a dash (`-`) in name

the-matt-morris commented 1 year ago

Checklist

[X] I added a descriptive title
[X] I searched open reports and couldn't find a duplicate

What happened?

I have the following environment.yml:

name: myenv
channels:
  - conda-forge
  - defaults
dependencies:
  # Python
  - python=3.9
  - pip
  - aiohttp>=3.8.3  # for some reason, poetry resolves to `3.7.4.post0`
  - scipy>=1.9.1
  - typer>=0.6.1
  - prefect>=2.8.1
  - h3>=3.7.4
  - pandas>=1.5.3
  - scikit-learn>=1.1.2
  - numpy>=1.24.1
  - python-xxhash>=3.0.0
  - dask==2022.11.1  # memory errors with `.read_parquet`
  - fastparquet>=2022.12.0
  - geopandas>=0.11.1,<1
  - osmnx>=1.3
  - networkx>=3
  - python-igraph>=0
  - pyproj>=3.4.0
  - pydantic>=1.10.2
  - pandera>=0.12,<1
  - pyarrow>=9.0.0
  - python-dotenv>=0.21.0
  - shapely>=2
  - gitpython>=3.1.29
  - dask-geopandas==0.3.0
  - pip:
    - sh
    - prefect-dask
    - libgeohash==0.1.1
    - s2sphere
    - numba
    - pandana

When I run conda-lock on this file, I get the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/conda_lock/src_parser/__init__.py", line 488, in seperator_munge_get
    return d[key]
KeyError: 'docker-py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/conda_lock/src_parser/__init__.py", line 491, in seperator_munge_get
    return d[key.replace("-", "_")]
KeyError: 'docker_py'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/conda-lock", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/conda_lock/conda_lock.py", line 1353, in lock
    lock_func(
  File "/usr/local/lib/python3.9/site-packages/conda_lock/conda_lock.py", line 1083, in run_lock
    make_lock_files(
  File "/usr/local/lib/python3.9/site-packages/conda_lock/conda_lock.py", line 408, in make_lock_files
    lock_content = lock_content | create_lockfile_from_spec(
  File "/usr/local/lib/python3.9/site-packages/conda_lock/conda_lock.py", line 801, in create_lockfile_from_spec
    deps = _solve_for_arch(
  File "/usr/local/lib/python3.9/site-packages/conda_lock/conda_lock.py", line 737, in _solve_for_arch
    pip_deps = solve_pypi(
  File "/usr/local/lib/python3.9/site-packages/conda_lock/pypi_solver.py", line 327, in solve_pypi
    src_parser._apply_categories(requested=pip_specs, planned=planned)
  File "/usr/local/lib/python3.9/site-packages/conda_lock/src_parser/__init__.py", line 502, in _apply_categories
    for dep in seperator_munge_get(planned, item).dependencies
  File "/usr/local/lib/python3.9/site-packages/conda_lock/src_parser/__init__.py", line 493, in seperator_munge_get
    return d[key.replace("_", "-")]
KeyError: 'docker-py'

FYI docker-py is a dependency for prefect

Conda Info

No response

Conda Config

No response

Conda list

No response

Additional Context

No response

maresb commented 1 year ago

Awesome, thanks for the report, this is super useful! I slimmed down your example to

channels:
  - conda-forge
dependencies:
  - prefect
  - pip
  - pip:
    - prefect-dask
platforms:
  - linux-64

and added it as a test in the branch gh-380, so hopefully this makes it easier for someone to look into it and open a PR with the goal of getting a particular test to pass. I'm guessing this is a problem with the conda-forge ⟷ PyPI lookup table.

Possibly related to:

253 "Potential package name confusion breaks conda-lock", possible duplicate
290 "Fix #253: package confusion between conda and pip", does this solve it?
364 "Refactor lookup._LookupLoader", it'd be great to get this merged, reviews very welcome!

maresb commented 1 year ago

Wow, this is great! Merging ~#364~ #290 into the gh-380 branch solves the issue. :rocket:

the-matt-morris commented 1 year ago

Awesome, thanks @maresb !!

maresb commented 1 year ago

Ah, no, sorry! I made a typo in my previous comment which linked the wrong issue. Unfortunately this is still open until we merge #290.

conda / conda-lock