[Feature Request]: Package with poetry

[happysalada]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

I'm proposing to manage dependencies with Poetry. It generates a lockfile making the environment reproducible accross machine. This would reduce the number of errors and bugs due to specific environment on particular machines. This would also make it easier to package this for linux based distributions.

Proposed workflow

Poetry would be for installation and generating virtual environments with all the dependencies https://python-poetry.org/docs/basic-usage/

Additional information

happy to provide more details if needed. I'm looking at packaging this for nixos, and if this used poetry, it would be much easier.

[akx]

I don't think Poetry will cut it here, since the dependencies vary between e.g. graphics card vendors and other things that can't be defined via environment markers.

[happysalada]

it looks like they are supported https://python-poetry.org/docs/dependency-specification/#using-environment-markers

[akx]

@happysalada I said "things that can't be defined via environment markers". There's no environment marker for "has an AMD GPU", etc.

[happysalada]

That makes sense, i missed that! Could a workaround be to define several packages ? One for amd gpu and one for nvidia ones ? I dont know if there would be too many variations or if that would be too much work

[BuildBackBuehler]

Would love to see where exactly this issue exists. I had just been attempting to do just this manually. I'm a noob so forgive me if this is too vague.

Let me start off with what is the simplest, most elegant solution in my mind. A Poetry-friendly setup.py would do the trick. As it stands, if one uses Poetry add git+https://url-to-automatic-1111.git it results in errors as I suppose Poetry cannot translate the pyproject.toml nor the requirements.txt/requirements_versions.txt without some help. But Poetry does check a slew of different files (like setup.py) to derive the necessary package requirements. I imagine if there was just a clear origin for Poetry, a copacetic pipeline could be integrated for it. All the tool dependencies can be laid out in this setup.py for any combo of Linux/Mac/Windows AMD/Nvidia/M1. Poetry doesn't even need to rely on .venv if that makes any difference.

Manually speaking, I just did Poetry init and then imported package dependencies via Poetry Import (which snags the requirements.txt) and cat requirements_versions.text | xargs Poetry add. Seemed to perhaps take issue with where to source these packages, but one can also add a source (in whatever format/whichever remote source) into this hypothetical Poetry base setup file. Also of note, I am using rtx (rust-based ASDF clone), but I just laid out the necessary versioning of Poetry/Python in its local config file. It does allow one to specify the target .venv to use if that's relevant, or there's Poetry Shell so Poetry can create/natively manage a virtual env.

Hope that helps! Or that I can be directed to the tangible breakpoints, or that this bump catches the attention of some Poetry enthusiast who actually knows what they're doing 😅

[Pluckerpluck]

I would like to add that I actively dislike poetry. Every since the dev refused to listen to the community and add .env file loading into the poetry shell command. Also, its primary benefit used to be dependency resolution, but that is now properly handled by pip.

If you did wish to lock versions more strictly, I would suggest instead using pip-tools, which simply takes your pip install, performs it dry, and saves every dependency and its exact version into a locked file. This locked file can then be pip installed normally, by anyone, without an end user requiring a third-party tool.

This obviously still suffers the same problem poetry is facing here though, which is that in order to lock for multiple environments, you need to be able to run the install in all of those environments. However in pip-tools's case, you simply create a number of locked requirements files as needed.

---

Anyway, yeah, as an avid fighter against poetry I had to share my thoughts here.

[happysalada]

Thanks, thats helpful ! I didnt know about pip-tools

[akx]

Closing this for now because I'm pretty sure this isn't going to happen :)