Use Conda Forge graph for package name mappings and less downloads

Quansight-Labs / beni

Generate conda environment.yml from PEP 621 and/or flit config.

MIT License

10 stars 1 forks source link

Use Conda Forge graph for package name mappings and less downloads #13

Closed flying-sheep closed 3 years ago

flying-sheep commented 3 years ago

Fixes #12

However, not everything seems to be in there, e.g. black is missing. Maybe the file only contains mappings that are not trivial?

saulshanabrook commented 3 years ago

Oh nice, this is awesome! @jaimergp do you know how to best use this file?

flying-sheep commented 3 years ago

I investigated a bit, but I can’t find a simple list of packages available. I think:

cf-graph-countyfair/master/graph.json contains all conda packages (and their for us useless dependency info)
cf-graph-countyfair/master/mappings/pypi/name_mapping.yaml contains all combinations of PyPI name, conda name, and import where the three are not equivalent
cf-scripts/master/conda_forge_tick/pypi_name_mapping_static.yaml the same as the one before, but manually maintained. IDK if the info in this one is included in the previous one.

jaimergp commented 3 years ago

cf-graph-countyfair/master/mappings/pypi/name_mapping.yaml contains all combinations of PyPI name, conda name, and import where the three are not equivalent

Yep this is the one you need. It should only contain known packages (to conda-forge) that don't share a name with PyPI. @viniciusdc might have some more info, since he's more involved with the bot infrastructure there.

viniciusdc commented 3 years ago

cf-graph-countyfair/master/mappings/pypi/name_mapping.yaml contains all combinations of PyPI name, conda name, and import where the three are not equivalent

Yep this is the one you need. It should only contain known packages (to conda-forge) that don't share a name with PyPI. @viniciusdc might have some more info, since he's more involved with the bot infrastructure there.

Indeed that's the best source for your problem. There is a notebook where you can inspect some of the metadata within the conda-forge json in that repo as well, if needed. About the usage of the static maps, this one pypi_mapping.py generates the magic

flying-sheep commented 3 years ago

Great, thank you!

This is looking good. Would be nice if there was a smaller JSON blob to download for the list of packages on conda forge, but with some caching, the graph.json will work too.

flying-sheep commented 3 years ago

Using it now looks like this (once an hour, otherwise no progress bars)

$ python -m beni pyproject.toml
Creating cache for name_mapping.yaml at /home/phil/.cache/beni: 100%|█████████████████████████████| 636k/636k [00:00<00:00, 5.46MB/s]
Creating cache for graph.json at /home/phil/.cache/beni: 100%|██████████████████████████████████| 8.28M/8.28M [00:01<00:00, 5.70MB/s]
channels:
- conda-forge
dependencies:
- ...

saulshanabrook commented 3 years ago

Awesome! That looks great 🎈