Micropip should be able to process requirements files

[hoodmane]

🚀 Feature

We should have a micropip version of pip install -r requirements.txt. Maybe micropip.install(requirements_file="some/path").

[rth]

We should probably nudge users toward using some form of a lock file instead of a requirement.txt file, but I suppose it still makes sense to support requirements.txt files for existing projects.

[rth]

BTW, tried recently pip-tools for lock file generation in a requirement.txt and was very happy with it.

[miohtama]

Any ideas/examples how this could be accomplished on the caller side today e.g. by feeding packages one by one?

• We are currently trying a library with complex dependencies getting installed through micropic
• The package is managed by poetry, so we can generate all kind of exports of requirements
• micropip installation currently fails, as some parts of micropip dependency resolution seem to be different/broken
• However it is not possible to tell, because at least to our knowledge there is currently not an easy to way to get micropip installation process give a meaningful log output what it is doing and why to address the issues
• Not allowing micropip to figure out dependencies on its own, but force feeding it a requirements.txt pin down list or similar should solve this issue

[rth]

micropip installation currently fails, as some parts of micropip dependency resolution seem to be different/broken

@miohtama Can you open a new issue with the details of the errors you are getting?

However it is not possible to tell, because at least to our knowledge there is currently not an easy to way to get micropip installation process give a meaningful log output what it is doing and why to address the issues

micropip has fairly rudimentary dependency resolution where it starts from the first package, loads recursively all requirements, and hopes that all will be OK. You can see what it is doing in the dev console log.

Not allowing micropip to figure out dependencies on its own, but force feeding it a requirements.txt pin down list or similar should solve this issue

Well, you can do something like,

from pathlib import Path

requirements = Path('requrements.txt').read_text().splitlines()
await micoropip.install(requirements)

the question is how to create that frozen list of requirements. What poetry or pip-compile will give you will not be correct, since it will not take into account binary wheels that exist for the emscripten/wasm platform. We are exploring this topic in https://github.com/pyodide/pyodide-lock/issues/10 but it's still early days.

Meanwhile, you can also take the output of frozen requirements given to you by those tools and manually tweak it until it works. In particular, any package version with binary wheels (e.g. numpy etc) need to match the version in the Pyodide distribution otherwise it will not be able to find the wheel. Though it's certainly not a very user friendly approach.

[joemarshall]

@rth pyodide build can output a flat requirements.txt / lockfile for a build already.

I do think it needs some fixup in terms of how one makes pyodide/ micropip aware of dependencies of wheels you've installed but that is a lot more straightforward (and I've put in an issue relating to this)

78