search

pyodide / micropip

A lightweight Python package installer for Pyodide
https://micropip.pyodide.org
Mozilla Public License 2.0
68 stars 16 forks source link

Feature: Dependency fixup for manually installed wheels #78

Open joemarshall opened 10 months ago

joemarshall commented 10 months ago

I'm building a package which has a large dependency tree. So I build it with pyodide build including dependencies, and get myself all the wheels (18 wheels, plus some pyodide dependencies). Then I script micropip to load all the wheels (with deps turned off because I've manually calculated them already) and call freeze to get a new JSON file.

So far so good. But the JSON file has no dependencies in it, and I have half an hour's work to manually edit every package and add in dependencies.

I guess I want some way that after loading a bunch of wheels I can call micropip.check_package_dependencies("package") and it will check and fix the dependencies. It could also report missing deps.

In theory I can use micropip.install in reverse order (starting from leaf nodes of the dependency tree) and leave dependency resolution on, but I'm a bit worried this will be fragile. I'd rather be able to just chuck all my wheels at it and then fix the dependencies.

Does this idea make sense?

ryanking13 commented 10 months ago

Thanks @joemarshall! I think we are discussing similar stuff in pyodide-lock repository. There were a few ideas with no conclusion, so feel free to comment there. Personally, I don't prefer adding complex dependency resolution stuff in micropip so I was thinking that it would be nice if we could reuse the pip's dependency resolver.

hoodmane commented 10 months ago

I think this idea makes sense.

joemarshall commented 9 months ago

@hoodmane @ryanking13

I sketched out a fix (#79) to micropip.freeze so it fixes up the dependencies between packages which are newly imported. This means that you can:

1) Build a package using pyodide build --with-dependencies which means you have the package and all dependencies for running it in pyodide available to you in a single directory. 2) Install all these packages manually using micropip.install(deps=False) (from a list, or a requirements.txt or whatever) 3) Call micropip.freeze 4) Get a valid pyodide-lock.json which you can use to have these packages available in your distribution with dependency resolution working.

It doesn't really do any dependency resolution (that happens offline in pyodide build) - it just looks at the list of packages in the wheel dependencies and compares against the frozen package list - if any of the wheel dependencies are in there, it adds them to the dependency list for that package.

I'm not sure whether or not micropip.freeze should exist at all, or whether it should be entirely superseded by pyodide-lock, but until it is deprecated, this makes it work in this case which is largely what people want to pay me to do with pyodide right now...