How do I make part of and auto-enable plugins in my project

[Maciej-Zwolinski]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

Issue

system: Ubuntu-20.04 poetry: 'master'

I need poetry-dotenv-plugin to be a part of my project (i need to set up environment variables to point to proper installation of CUDA, and don't want the user to have to do that manually or modify .bashrc in any way). The documentation is silent on how to make a plugin part of the project's dependency - simply adding poetry-dotenv-plugin = '*' in pyproject.toml's dependencies section and running poetry run python3 -c 'import os; print(os.environ)' does not return modified environment. What is an expected syntax here?

[Secrus]

Hi. There is info in docs about how to add plugins to your poetry instance: https://python-poetry.org/docs/master/cli/#self-add. Could you confirm that it works for you?

[Maciej-Zwolinski]

Hi, I want the poetry install command to add plugins specified (how, I do not know, that's what I'm asking) in my pyproject.toml.

Is that possible? Otherwise a user would need to run poetry install && poetry self add poetry-dotenv-plugin to be able to fully utilize the package.

[Secrus]

I think you don't understand how plugins work. Plugins are not part of the package you are making, they are for your Poetry instance. You add a plugin once and it is available in your Poetry instance across all your Poetry projects on that machine.

[Maciej-Zwolinski]

Ok, thank you, that's very helpful.

Is there a way to define plugins in a config of some sort so they can be enabled with a single poetry command?

[danielknell]

looks like the PoC in https://github.com/python-poetry/poetry/pull/5740 would solve this?

[Kobold]

@Maciej-Zwolinski Would love to see this too!

Plugins are not part of the package you are making, they are for your Poetry instance.

@Secrus The use case I, @Maciej-Zwolinski, and I suspect others are looking address are workflow issues across all developers and all Poetry instances across a project. Management of common tasks (nat-n/poethepoet), common environments (mpeteuil/poetry-dotenv-plugin), precommit (vstrimaitis/poetry-pre-commit-plugin) and such are things individual developers may want to customize — but most projects want to have shared common ways to address these needs.

The deeper reason why poetry and yarn are so much better to use than pip and early npm is that they acknowledge that development environment instances are transient, and that bulletproof reproducibility across instances is really valuable for teams of any size. Therefore, I would suggest Poetry find ways to address developer needs while discouraging per-Poetry-instance customization. If Poetry encourages per-instance customization, you're going to find Github issues popping up about how to keep Poetry instances in sync — recapitulating exactly the issues that Poetry was meant to address.

Re: "plugins". Again, it's fine if Poetry wants per-instance customization, but the nascent ecosystem is already confused about the intent of this API. The three most starred plugins in the Github poetry-plugins tag:

• mpeteuil/poetry-dotenv-plugin (35 stars) - "A Poetry plugin that automatically loads environment variables from .env files into the environment before poetry commands are run."
• DavidVujic/poetry-multiproject-plugin (15 stars) - "A Poetry plugin that aims to simplify package & dependency management for projects in Monorepos."
• DavidVujic/poetry-polylith-plugin (10 stars) - "A Poetry plugin that aims to simplify working with Polylith monorepos."

These are all tools and capabilities that poetry users probably want in common for all developers on their repository, not just on one-off instances of the repository. I would suggest Poetry take care in thinking about what use cases plugins are for.

looks like the PoC in https://github.com/python-poetry/poetry/pull/5740 would solve this?

@danielknell it definitely does solve this but I would suggest implementing a version of npm run in Poetry as a lightweight intermediate solution. It does seem like the PoC is a significant philosophical divergence from the statement "I think you don't understand how plugins work. Plugins are not part of the package you are making, they are for your Poetry instance." Given that philosophical conflicts take time to resolve, an npm run equivalent may get us 80% of the way there with less friction?

In terms of an npm run being able to solve these issues without shared plugins, I've been using poe as a dev dependency (poetry add --with dev poethepoet) then including example commands in the README:

poetry run poe task tox
poetry run poe task pre-commit
# Have these python task invocations use `dotenv` type functionality internally:
poetry run poe task ipython
poetry run poe task python ...

And asking users to ignore the poetry run poe task noise, or alias it away. This achieves exactly the declarative dependencies + shared command definitions the original issue looks to address.

I would love it if some poethepoet functionality (not the full shebang) was upstreamed to Poetry, so the above could just be:

poetry task tox
poetry task pre-commit
# Have these python task invocations use `dotenv` type functionality internally:
poetry task ipython
poetry task python ...

or something! Just food for thought.

[neersighted]

The tasks system (chosen because scripts already has a well-known meaning in Python) is something that has been discussed, and while originally considered out of scope for Poetry, reconsidered due to popular demand. Note that scripts does let you do this with Python files, but due to user confusion over how to make use of this functionality (as well as the fact that it will be exposed to consumers of packages, if you distribute them), has proven to not be the best fit.

That being said, the tasks feature (or similar) is orthogonal to what is being discussed here, which is repo-level plugins.

Plugins are not part of the package you are making, they are for your Poetry instance.

This statement is a summary of how things are. Poetry plugins are currently global as the API is experimental/a MVP, and discovery of custom entrypoints in the environment of a package is well-defined, stable, and easy from a design perspective.

5740 is one exploration of how we can extend project-level plugin support, but currently has a lot of hurdles to overcome, from both a design and implementation standpoint, before it can be considered for merge. Nonetheless, it is intended that Poetry will eventually support both global-level plugins (e.g. for developer ergonomics, or optional global use like the dotenv plugin, which may be an individual developer's preference and not something enforced by a project) and project-level plugins (e.g. for a setuptools_rcm equivalent).

Until then, the best way to handle this is like any other developer tool currently -- add it to your project's dependency graph. This has the effect of increasing resolve times, surface area for conflicts, and potentially preventing you from upgrading packages, but will be stable and 'it just works' today (assuming that you activate the Poetry-managed environment or poetry shell so that you use the 'inner' Poetry). It's a clunky method, but project-level plugins is almost as complex as the plugin feature itself and will take time to design and develop.

[Kobold]

but will be stable and 'it just works' today

💯

That thinking makes complete sense. I'm glad to hear it and thank you for taking the time to respond. Poetry has solved so much Python package pain in my life. Thanks for all your effort in making it happen.

[alanwilter]

Ok, I understand the point but it would still be nice to do poetry install and get my plugins right out of the box. Because I don't see the point of having to do myself poetry self add poetry-bumpversion in every new project or git clone I do, if using other computers or sharing a repo with other developers.

[DaStoll]

Would absolutely love this feature!

[yawor]

Maybe create some sort of a plugin manager plugin. It would still need to be installed manually once on every machine you use, but then, for example during the install command, it could check some custom field in the pyproject file and install all required plugins if they're not yet installed. They'd still be installed globally for the poetry instance. Not ideal, but at least you could just add another plugin to the project and just run poetry install again everywhere.

[sliedes]

Is there at least a way to make poetry fail a build if a certain plugin is not installed?

I don't know if poetry-grpc-plugins is a reasonable way to handle generating code (grpc/protobufs); perhaps it's somehow best handled differently, as I just started to look into this. But if that plugin is used to generate code and there is no way to specify that a plugin is required, the end result is that any user will get a mysterious import error instead of a helpful error message telling what they need to do.

[bswck]

Bump on that one.

[maxdeichmann]

+1 would love to see this feature

[jwueller]

First impressions from the perspective of an experienced developer that's dealing with Python for the first time (i.e me):

The experience is, frankly, atrocious, compared to almost any other dependency management system I've ever used. Even dependency hell in Maven is more fun than this. I want one command that reliably and correctly builds my package. Anything that requires someone to add extra steps means that people will get it wrong, probably wasting an unfathomable amount of time across all poetry users in troubleshooting and confusion.

What's the point in even having plugins if you can't rely on any of them? Making anything in the build rely on the local system configuration, apart from the absolute minimum baseline tools being installed, is just unreasonable. The tools are supposed to make the process better, not create additional and arbitrary hoops to jump through. A tool exists to serve the users, not the other way around.

What happens in practice is that someone puts a build.sh script in the project that has to dependency manage the dependency management tool.

There seems to be a fundamentally flawed development philosophy across this whole ecosystem. As an example: One suggestion in this thread was to add a plugin to make plugins usable by adding yet another hoop to jump through, using the same broken-by-design system that got us here in the first place. Please stop piling workarounds on top of each other and actually address the underlying problems. (I'm sorry to the author of that particular comment, it's not intended as a personal attack!)

I would seriously urge you to move determinism (i.e. reproducibility and correctness), as well as developer experience all the way to the top of your priority list, since it's the most important job of a tool like this.

Maybe this isn't a poetry limitation, but a more fundamental one. I have a hard time even understanding the organizational tangle of who makes these kinds of decisions for a project like this. At this point I'm just at a loss for words.

I apologize for my confrontational tone. I am a very frustrated first-time poetry user.

[yawor]

There seems to be a fundamentally flawed development philosophy across this whole ecosystem. As an example: One suggestion in this thread was to add a plugin to make plugins usable by adding yet another hoop to jump through, using the same broken-by-design system that got us here in the first place. Please stop piling workarounds on top of each other and actually address the underlying problems. (I'm sorry to the author of that particular comment, it's not intended as a personal attack!)

I think you've meant my comment. Don't worry, I'm not offended or anything. I actually agree with you regarding this issue. Take a look at Hatch. Their plugin support is much better and actually allows to add and configure plugins in pyproject.toml. The only thing that keeps me from switching some of projects I work on to Hatch is that they don't (yet) support installing local dependencies as editable and I need that functionality.

[alanwilter]

@yawor

The only thing that keeps me from switching some of projects I work on to Hatch is that they don't (yet) support installing local dependencies as editable and I need that functionality.

Thanks for that. I'm really considering the options, specially now that Astra made uv and took over rye. But your remark is also critical for me.

[radoering]

There is #9547 now, which is a complete overhaul of the plugins part of the Poc in #5740 and ready for review.