search

pyodide / micropip

A lightweight Python package installer for Pyodide
https://micropip.pyodide.org
Mozilla Public License 2.0
68 stars 16 forks source link

Micropip "dependency" resolution in a Python-only runtime #104

Open flxmr opened 2 months ago

flxmr commented 2 months ago

So, I started using panel and would like to deploy something with their pyodide-export. As I don't want to rely on the various CDNs, I want to get all the wheels to host them myself. As there doesn't seem to be a canonical requirements-resolver in the python world (...) and pyodide-packages are not yet integrated with Pypi, I think this would eventually result in replicating micropip functionality.

I gather this is essentially what micropip.freeze does - solving this would thus also solve these problems I'd gather https://github.com/pyodide/micropip/issues/68 https://github.com/pyodide/micropip/issues/78. I think pyodide-pack does this too - but it uses Javascript. Or is it pyodide-lock?

So: would you be open to make micropip a) importable in any python, so that b) you can just run the (primitive, but better than replicated...) resolver in python to spit out a dict of wheels? c) any ideas how to start? The things which are confusing me a lot: where are the pyodide-CDN-locations injected?, which are these? where is this documented?

So, looking into pyodide-lock, people are open to this solution: https://github.com/pyodide/pyodide-lock/issues/8

ryanking13 commented 2 months ago

Hosting packages is not very hard. You can download all wheels in the Pyodide distribution, host it somewhere (that supports https://peps.python.org/pep-0691/), and update the index URL used in micropip (micropip.set_index_urls)

The tricky parts are optimizing the dependency resolution. I think pyodide-lock should be able to create a lockfile from scratch, but for now in only supports adding wheels into the existing lockfile. So users need to manually modify the URLs in the lockfile.

So I think there are a number of features that could be put into pyodide-lock to improve this, but I'm not sure what would satisfy everyone. And in the long term, I think a lot will change when it becomes possible to upload WASM packages to package indexes like PyPI.

any ideas how to start?

I think we can start with connecting micropip and pyodide-lock, so that micropip.freeze can create a pyodide-lock compatible lockfile (It may work, but we don't have tests).

Then maybe we can add more features in pyodide-lock for modifying lockfiles, so that, for example, making CDN locations can be changed.