Saving custom scripts in pyproject.toml (alias npm run)

[bersace]

• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have searched the documentation and believe that my question is not covered.

Feature Request

What do you think of having a scripts described in pyproject.toml ? Something like Procfile or script section in packages.json.

Here is a suggestion of configuration & usage:

[tools.poetry.proc]
build = "python manage.py collectstatic"
test = "pytest tests/unit/"
serve = "flask run"

$ poetry proc build
...
$ poetry proc serve
...

I think it would help project to not require Makefile nor tox to share commands and in the same time, ensure commands are properly executed in selected environment.

What do you think of this ?

[SanketDG]

See https://github.com/python-poetry/poetry/issues/2310 for a broader spec.

[bersace]

@SanketDG I think it's something else.

• scripts is about shipping CLI entrypoints to final users. Think /usr/bin/command
• proc is about sharing recipes or command alias to developers. Think tox -e unit

[abn]

This has been discussed in #591 which was closed with https://github.com/python-poetry/poetry/pull/591#issuecomment-504762152.

As was stated there, this might be a candidate feature for a plugin once we have that baked into poetry >=v1.1.0.

[bersace]

@abn yes, that's exactly the subject of this PR :-) A plugin is a good fit for this feature.

[nat-n]

@bersace in the mean time I'd suggest checking out a project I'm working on that solves this problem in a slightly different way

Poe the Poet a task runner that works well with poetry.

It provides a flexible system for defining tasks in the pyroject.toml and executing them with a dedicated executable. Basic usage looks like:

Example configuration with different task types:

[tool.poe.tasks]
test       = "pytest --cov=poethepoet"                                # simple command based task
mksandwich = { script = "my_package.sandwich:build" }                 # python script based task
tunnel     = { shell = "ssh -N -L 0.0.0.0:8080:$PROD:8080 $PROD &" }  # shell script based task

Example invocation:

# extra argument is passed to the underlying pytest command
poe test -v   

# If poethepoet is added as a dev-dependency
poetry run poe tunnel

And it supports zsh command completion!

EDIT: as of v0.12.0 it can also be used as a poetry plugin

[stranger-danger-zamu]

Leaving my 2 cents here on the whole "scripts in pyproject.toml" conversation. I understand that this will probably not going to be implemented or accepted, but at least I added my 2 cents.

---

This is a clearly desired feature and has had well supported PRs demonstrating community support for it going back a few years.

While I do get the maintainers wanting this sort of functionality outside of poetry's core codebase and scope, people are asking for it because they see poetry as Python's primary DX/DI. The birds eye view of Python's developer experience at the moment is that pyproject.toml will be the future of Python and poetry is in the position as the developer interface for pyproject.toml projects because it's the most full featured tool that uses it that also builds.

Taking a step back, I personally just want a tool that I allows me to have a known procedure for all (modern) python projects.

Rust has this with cargo where the requisite to get up and running to contribute to a project is cloning the repo, having rust and cargo, making some changes, and running cargo build.

Go has something similar (if not a little messy due to older codebases possibly not supporting modules). That said it's simple in those languages since at most, they statically compile their artifacts and are done in terms of bundling their dependencies with their artifact.

Meanwhile in Python-land, we might build a wheel with explicit python dependencies tied to hashes. But that might not bundle all the true dependencies (eg. system libraries).

So, we have to reach out for other artifact formats which do allow us to encapsulate all our dependencies (eg. Docker, deb, rpm, conda, OS golden images, etc). This is a bigger problem with Python packaging that is definitely outside of poetry's scope.

There are plenty of tools which can easily encode the procedures to bundle python wheels into a new artifact, but nine times out of ten they are excessive or they make poetry redundant and/or useless.

• Make is amazing. It allows developers to define complex graphs of dependencies and regenerate files based on modified time. However, Python projects aren't hugely complicated task graphs, they tend to just be linear dependencies making most of make's feature set more of a liability than a boon. For most python projects, you can just do the same thing with shell scripts with less cognitive overhead.
• Python task runners are great and many of them have wonderful features. But they are another thing you have to install and learn. They also can be replaced with a shell script with less cognitive overhead.
• Poetry Plugins share the same problem as python task runners. You have to install them separately and learn them separately. Even knowing that it is straight forward (not really since there is no official 1.2 release), many people can't reliably use this. These plugins can be made by anyone and, being honest, I doubt anyone is going to look at that codebase to check that it's not a security problem. I'm much more confident that poetry would be looked at instead of a random plugin by some person. You have to be cynical about security.
• So what about those shell scripts? Well for one you can't assume that they are cross-platform (thanks Windows!), so you either have to have more than one script for the appropriate platforms (making maintainability a bit of a chore) or add system dependencies for developing but that makes it intractable for some users (yay, enterprise security policies...).
• Technically, we have Python, so python scripts? This does work, however it can be a bit hairy to do some stuff (namely piping commands so for beginners it might be a bit much) and there is no spec or standard so get ready for arbitrary script names that will be stored arbitrarily (of course, people will not be insane and scripts will probably house everything, but people will always exceed your expectations). However, pyproject.toml seems to be swallowing all extra files that aren't your actual code which is great but then that means any extra script (python or otherwise) is swimming upstream.

To be honest, what I don't want is having to write documentation on how to set up a dev environment including on installing N plugins before they clone down the repository. Writing python scripts to run "tasks" might work but it can quickly become a soup of arbitrary code (and probably bad security practices). And if I am going through all that effort, I might as well just use setuptools or flit if poetry doesn't really have differentiating utility built-in from them.

It just feels a bit odd. If poetry is a package manager, then it shouldn't also build packages. You don't use APT or DNF to build deb or rpm files.

[Dilski]

This feels like a good candidate for a poetry plugin?

[neersighted]

https://github.com/nat-n/poethepoet and https://github.com/illBeRoy/taskipy both exist to do this, and Poe has a Poetry plugin (taskipy will likely gain one)

[Dilski]

Oh perfect, thank you!

[brunolnetto]

I just wrote an issue on the subject, but you are ahead of me. Linguistics call these words aliases. You must be familiar with it yourselves on Git environment and setup file .git/config.

If it is not already shipped, I suggest the following syntax:

 $ poetry alias "hocus-pocus" "echo Abracadabra" 
 $ poetry hocus-pocus
 Abracadabra

Main drawback: poetry alias introduces on the very most some code management "fear" of "malicious command". It evokes the question: who would write such a malicious command against itself? Nevertheless, a conscious peer-review vanishes such intents away.