Export requirements.txt

[maresb]

New issue to track the follow-up discussion from https://github.com/synapticarbors/pixi2ces/pull/3

Highlights:

With regard to the requirements.txt, is the idea that you'd generate the lock file and the requirements.txt, create the conda env with the lock file and then have a second step where you pip installed the requirements.txt?

Exactly.

Looking at what rattler_lock has for the PypiPackage https://docs.rs/rattler_lock/latest/rattler_lock/struct.PypiPackage.html it seems like we should just be able to copy the url into a requirements.txt, but I'd have to familiarize myself what it might look like if there were editable and how to handle that

I'll have to look into where the hashes are specified, but assuming they are easily accessible, I think it should be easy to do:

1. Packages with url + hash

2. Editable packages prepended by `-e`

I'll have to look at how pixi specifies git dependencies, but the data should be in there: https://pixi.sh/v0.22.0/reference/configuration/#full-specification

I think I see, so it's basically a matter of translating rattler's structs into a requirements.txt line? If that's the case, I think most of the requirements spec is spelled out here. So for editable, you just prefix the line with -e. It would be good to have hash-checking mode (suffix --hash=sha256:deadbeef...) ~and extras (packagename[extra])~. It looks like we don't have to deal with "requirement specifiers" since those are essentially predetermined by the platform and the locked version of Python.

There's the issue of credentials for private repositories, but let's worry about that later.

[synapticarbors]

I was playing around with this a little, using the project in https://github.com/prefix-dev/pixi/tree/main/examples/pypi-source-deps since that seems to cover a lot of ways of specifying pypi dependencies. In that case, the urls that are in pixi lock are not all compatible with pip, namely direct+, which must be some sort of rattler specific marker. It works if you strip that off. With some new code, it now produces a requirements.txt containing the following for the pypi dependencies:

https://files.pythonhosted.org/packages/bb/2a/10164ed1f31196a2f7f3799368a821765c62851ead0e630ab52b8e14b4d0/blinker-1.8.2-py3-none-any.whl --hash=sha256:1779309f71bf239144b9399d06ae925637cf6634cf6bd131104184531bf67c01
https://files.pythonhosted.org/packages/1c/d5/c84e1a17bf61d4df64ca866a1c9a913874b4e9bdc131ec689a0ad013fb36/certifi-2024.7.4-py3-none-any.whl --hash=sha256:c198e21b1289c2ab85ee4e67bb4b4ef3ead0892059901a8d5b622f24a1101e90
https://files.pythonhosted.org/packages/2e/7d/2259318c202f3d17f3fe6438149b3b9e706d1070fe3fcbb28049730bb25c/charset_normalizer-3.3.2-cp312-cp312-macosx_10_9_x86_64.whl --hash=sha256:ddbb2551d7e0102e7252db79ba445cdab71b26640817ab1e3e3648dad515003b
https://github.com/pallets/click/releases/download/8.1.7/click-8.1.7-py3-none-any.whl
git+https://github.com/pallets/flask@a8956feba1e40105e7bc78fa62ce36c58d1c91e1
https://files.pythonhosted.org/packages/e5/3e/741d8c82801c347547f8a2a06aa57dbb1992be9e948df2ea0eda2c8b79e8/idna-3.7-py3-none-any.whl --hash=sha256:82fee1fc78add43492d3a1898bfa6d8a904cc97d8427f683ed8e798d07761aa0
https://files.pythonhosted.org/packages/ef/a6/62565a6e1cf69e10f5727360368e451d4b7f58beeac6173dc9db836a5b46/iniconfig-2.0.0-py3-none-any.whl --hash=sha256:b6a85871a79d2e3b22d2d1b94ac2824226a63c6b741c88f7ae975f18b6778374
https://files.pythonhosted.org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl --hash=sha256:c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef
https://files.pythonhosted.org/packages/31/80/3a54838c3fb461f6fec263ebf3a3a41771bd05190238de3486aae8540c36/jinja2-3.1.4-py3-none-any.whl --hash=sha256:bc5dd2abb727a5319567b7a813e6a2e7318c39f4f487cfe6c89c6f9c7d25197d
https://files.pythonhosted.org/packages/42/d7/1ec15b46af6af88f19b8e5ffea08fa375d433c998b8a7639e76935c14f1f/markdown_it_py-3.0.0-py3-none-any.whl --hash=sha256:355216845c60bd96232cd8d8c40e8f9765cc86f46880e43a8fd22dc1a1a8cab1
https://files.pythonhosted.org/packages/48/d6/e7cd795fc710292c3af3a06d80868ce4b02bfbbf370b7cee11d282815a2a/MarkupSafe-2.1.5-cp312-cp312-macosx_10_9_x86_64.whl --hash=sha256:3c6b973f22eb18a789b1460b4b91bf04ae3f0c4234a0a6aa6b0a92f6f7b951d4
https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl --hash=sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8
https://files.pythonhosted.org/packages/88/5f/e351af9a41f866ac3f1fac4ca0613908d9a41741cfcf2228f4ad853b697d/pluggy-1.5.0-py3-none-any.whl --hash=sha256:44e1ad92c8ca002de6377e165f3e0f1be63266ab4d554740532335b9d75ea669
https://files.pythonhosted.org/packages/f7/3f/01c8b82017c199075f8f788d0d906b9ffbbc5a47dc9918a945e13d5a2bda/pygments-2.18.0-py3-none-any.whl --hash=sha256:b8e6aca0523f3ab76fee51799c488e38782ac06eafcf95e7ba832985c8e7b13a
git+https://github.com/pytest-dev/pytest.git@6c806b499ddbb844753b5c8c4d70a8b98b9d1c3a
git+https://github.com/psf/requests.git@0106aced5faa299e6ede89d1230bd6784f2c3660
https://files.pythonhosted.org/packages/87/67/a37f6214d0e9fe57f6ae54b2956d550ca8365857f42a1ce0392bb21d9410/rich-13.7.1-py3-none-any.whl --hash=sha256:4edbae314f59eb482f54e9e30bf00d33350aaa94f4bfcd4e9e3110e64d0d7222
https://files.pythonhosted.org/packages/ca/1c/89ffc63a9605b583d5df2be791a27bc1a42b7c32bab68d3c8f2f73a98cd4/urllib3-2.2.2-py3-none-any.whl --hash=sha256:a448b2f64d686155468037e1ace9f2d2199776e17f0a46610480d311f73e3472
https://files.pythonhosted.org/packages/9d/6e/e792999e816d19d7fcbfa94c730936750036d65656a76a5a688b57a656c4/werkzeug-3.0.3-py3-none-any.whl --hash=sha256:fc9645dc43e03e4d630d23143a04a7f947a9a3b5727cd535fdfe155a17cc48c8
-e ./minimal-project

The changes to generate the file are in https://github.com/synapticarbors/pixi2ces/pull/5. I've done some testing on that one example project and everything seems to work including uv pip install -r requirements.txt --verify-hashes.

pip allows for a lot of variants for defining packages, so I'm not sure if I've covered all of the possibilities. I guess it's probably time to start developing some tests for this project.

[maresb]

Ooh, this is really awesome!!!

One desirable property of such a list is for it to be "topologically sorted" meaning that if you install from top to bottom then at each point the dependencies are satisfied. (If your lockfile contains the dependency information then you can store things in alphabetical order and topologically sort them yourself before install.) In contrast, your list looks alphabetically sorted. I guess probably doesn't matter much since most modern tools will sort automatically, but toposort would theoretically allow for a fast offline install.

[synapticarbors]

Currently I'm just writing them in the order that they are specified in the lockfile packages struct. I poked around the pixi source code a bit, but haven't found if they are doing a topological sort at any point, although it's possible I just didn't stumble across it. I'm not inclined to write a topo sort from scratch, but if there is functionality in rattler or pixi that could be used, I'd be happy to add it.

[synapticarbors]

@maresb -- Just a heads up that I've submitted a PR to upstream all of the functionality in pixi2ces into pixi: https://github.com/prefix-dev/pixi/pull/1873

[maresb]

Wonderful!!!