Add support for `scripts`

[ojii]

It would be nice if poetry supported the scripts feature of setuptools (note: this is not the same as entry points).

I tried using a custom build script but it seems to be ignored by poetry when building wheels.

[cauebs]

So, this already works:

[tool.poetry.scripts]
funniest-joke = "funniest.command_line:main"

Maybe we could support this as well:

[tool.poetry.scripts]
funniest-joke = { path = "bin/funniest-joke" }

...analogous to how we can specify dependencies. @sdispater what do you think?

[ojii]

Thanks for the info, I tried a scripts key under [tool.poetry]. I either missed that in the docs or it isn't there (if that's the case, I'll try to open a PR tomorrow).

[cauebs]

It's documented here: https://poetry.eustace.io/docs/pyproject/#scripts But I often miss these things as well :P

[ojii]

Ah I misread your initial comments. What's called "scripts" in poetry is called "entry points/console scripts" in setuptools. What I'm referring to is the "scripts" keyword to setup(...) which lets you specify things files as scripts (as opposed to python modules + functions). So it's the second example in your first comment I'm interested in.

If at all possible, if those scripts could be somehow selected per-platform, that would be amazing (maybe only for platform-specific wheels?)

[ojii]

For those interested, I'm working on this at https://github.com/ojii/poetry/tree/non-python-scripts

[cauebs]

@ojii I was just testing you fork and it appears to be working, but I think the syntax I proposed is misleading unless you rename the script, which is something setuptools doesn't do.

Say I have this in my pyproject.toml

[tool.poetry.scripts]
cli = { path = "foo/bar" }

I might expect to be able to run cli, but only bar will work. You could either rename the script to whatever is specified, to maintain the same syntax, or we can change the syntax, but I can't think of anything.

[ojii]

You could either rename the script to whatever is specified, to maintain the same syntax, or we can change the syntax, but I can't think of anything.

I think it's reasonable to just reject those scripts if the filename does not match the key. (In setuptools, it's a list, not a mapping).

[markovendelin]

I think it's reasonable to just reject those scripts if the filename does not match the key. (In setuptools, it's a list, not a mapping).

Not sure that such rejection would be reasonable. As a user, I would expect either a list of scripts (for plain copy) or mapping with renaming. Having to enter mapping with the same data entered twice (key and part of the value) does seem to be just a source for errors on user's side.

[ojii]

@cauebs @markovendelin I've updated my branch to re-name the scripts according to the users configuration. Should I open a PR or are there other blocking issues with this?

[cauebs]

Pretty cool! Worthy of a PR, in my opinion.

[rcisterna]

Are there plans to merge #304 any time soon? I've been recommending poetry in the company I work for, but sadly, this feature is a must have :(

Edit: Maybe there is a workaround that @cauebs or @sdispater can recommend for this in the meanwhile.

[Kamforka]

Is the script section able to execute commands that can help you integrate developer tasks into poetry?

I imagine it like this:

[tool.poetry.scripts]
test = "pytest tests/ --verbose"

It would be a nice feature to have, what do you think?

[cw-andrews]

Is the script section able to execute commands that can help you integrate developer tasks into poetry?

I imagine it like this:

[tool.poetry.scripts]
test = "pytest tests/ --verbose"

It would be a nice feature to have, what do you think?

I definitely agree. I am trying to do the same thing to run a molten app via gunicorn and it would be much easier to do it this way then write a script for a custom gunicorn application.

[purificant]

I'd like to be able to type poetry test and have that run a longer script with test configuration, similar to poetry build. poetry lint would be nice too, this could be done through 'scripts' if they were similar to https://docs.npmjs.com/misc/scripts where you can define arbitrary script names which perform context aware commands.

[mattyclarkson]

I'm keen for something similary to NPM scripts. I would like to use poetry as my full workflow tool so:

poetry format
poetry lint
poetry build
poetry test
poetry coverage

[germn]

Just wanted to share messy workaround I ended up with.

pyproject.toml

[tool.poetry.scripts]
pytest = 'scripts:pytest'

You'll need to create file scripts.py to delegate stuff:

scripts.py

import sys
import subprocess

def __getattr__(name):  # python 3.7+, otherwise define each script manually
    name = name.replace('_', '-')
    subprocess.run(
        ['python', '-u', '-m', name] + sys.argv[1:]
    )  # run whatever you like based on 'name'

Then run something, for example:

poetry run pytest --help

It works :)

[mm-matthias]

I'd also like to see a feature like this, currently I am doing

[tool.poetry.scripts]
generate-code="codegen:generate_code"

But this will also create the generate-code script when installing the package in non-dev mode.

To run scripts only in dev mode I propose this:

[tool.poetry.dev-scripts]
generate-code="codegen:generate_code"

This is analogous to tool.poetry.dependencies/ tool.poetry.dev-dependencies.

[brycedrennan]

@mm-matthias @germn @cw-andrews @Kamforka @purificant

It looks like there is a desire for developer/project specific helper scripts, but that's not what [tool.poetry.scripts] is for. As mentioned, scripts are entrypoints into a python packages. This section's purpose is for libraries to install useful command line tools. Libraries like pytest, poetry, and the aws cli would use this scripts section so you can call their tool from the command line.

Scripts in this section will be globally available in any python environment that installs that package. For example, if someone were to install a package that had

[tool.poetry.scripts]
pytest = 'scripts:pytest'

in its pyproject.toml, that would be overriding the pytest command everywhere in the environment.

For a lot of what is being described, which is project specific developer scripts, a Makefile sounds more appropriate. Makefiles have the advantage of being language independent and already installed on most systems. This means for example, your makefile could have a script that setup a installed poetry and used it to create a virtual environment.

Here is a simple example of a Makefile. Here is a more fleshed out one from one of my projects.

test:  ## Run the tests in current environment
    pytest

help: ## Show this help message.
    @## https://gist.github.com/prwhite/8168133#gistcomment-1716694
    @echo -e "$$(grep -hE '^\S+:.*##' $(MAKEFILE_LIST) | sed -e 's/:.*##\s*/:/' -e 's/^\(.\+\):\(.*\)/\\x1b[36m\1\\x1b[m:\2/' | column -c2 -t -s :)" | sort

In your project you could then run make test or make help.

I'd rather not have poetry turn into yet another way to run scripts for developers.

[brycedrennan]

Also you can see an example Makefile in the poetry codebase itself: https://github.com/sdispater/poetry/blob/master/Makefile

[jgirardet]

I'm waiting the plugin system to see if something like could'nt be added outside of poetry "core". Make file style command would be very usefull inside the pyproject.toml

[brycedrennan]

Looks like a similar feature is being added: https://github.com/sdispater/poetry/pull/591

[Kamforka]

@brycedrennan I see your point, but I still feel like this feature has a direct place inside the poetry ecosystem.

Also for those who've got used to setuptools like entry_points it's a bit confusing to use a section like scripts for the same purpose.

It makes it even more confusing for those who are somehow related to npm projects where the scripts section is used to declare tasks.

But as far as I know the dev-scripts section is already under construction to achieve similar functionality as npm-scripts.

[brycedrennan]

Naming aside, we shouldn't overload a section to handle both entrypoints and dev-scripts. And certainly not in a way that would hose the environments of people who installed our packages with global overrides of pytest or test.

I'll still have to have a Makefile to handle installing the proper python version (via pyenv) and poetry itself. That's why this just feels like unnecessary fragmentation. It also doesn't help for projects that use multiple languages. Makefile works for everything.

I concede there is demand for it though.

[Kamforka]

I didn't say we should overload, rather I was talking about a section that could handle task running.

I understand that basically anything can be achieved using makefiles, but honestly I never had a Makefile inside any of my Python projects neither seen one in projects I use as dependencies.

Also I think that the main ambition behind poetry is to make a proper solution to manage, build and package Python projects, so I think multi-language support is not a top priority here. However if one needs to support multi-language projects then makefiles are a nice fallback to rely on.

[brycedrennan]

I thought using the already existing scripts section to support developer tasks was what was proposed here. Based on what you're saying now, I'll interpret that previous comment to mean you wanted entrypoints to be moved to a different section.

Makefiles can be pretty messy in some projects and thus its common to find them intimidating. Here are makefiles from libraries you use:

• poetry
• flask
• flask-sqlalchemy
• psycopg2
• cookiecutter

[cauebs]

@brycedrennan I think you misunderstood what's being proposed here. setuptools's scripts are very close to entry-points. They are packaged with everything else and should be exposed to whoever installs the package. The difference is that, instead of specifying the path to a module and a function, you specify a file to be directly executed.

A feature more analogous to what you'd have with a Makefile is introduced in #591, possibly named tasks or dev-scripts.

[brycedrennan]

@cauebs I see what your saying, and I may be misunderstanding. Correct me if I'm wrong, but using the setuptools scripts for things like test = "pytest tests/ --verbose" would be a misuse no? Nor does setuptools scripts map to wanting something like poetry lint. Both of these are mentioned above.

[Kamforka]

@brycedrennan Yeah I was too easy on none of my the dependencies use Makefiles, and also I mixed the original proposal with the implementation of #591, so I apologize for that.

[germn]

Embarrassed to say, but I just didn't know Makefile can do such stuff. Always thought about it as about some tool exclusively for compiling. I see it now it can handle everything I wanted.

@brycedrennan thank you very much for pointing to it!

[purificant]

Hi @brycedrennan. Thank you for your suggestions.

I understand that you like Makefile, and I must agree that it is a very powerful tool that has been around for over 40 years. It is reasonably popular in the C/C++ world and the file format has been developed decades ago to solve common use cases around building and installing software binaries.

I think that what we are asking for here is different from what you are suggesting.

We :heart: Poetry.

We love Poetry because of it's beautiful goal: Python dependency management and packaging made easy. We love Poetry because it comes with all the tools you might need to manage your projects in a deterministic way. We love Poetry because it lets us easily build and package our projects with a single command. We love Poetry because we can make our work known by publishing it to PyPI. We love Poetry because having an insight into our project's dependencies is just one command way.

If you notice, a lot of the above can be achieved by a combination of other tools, including Makefile. But we :heart: Poetry.

To quote the original creator, Sébastien Eustace:

I built Poetry because I wanted the one tool to manage my Python projects from start to finish. I wanted something reliable and intuitive that the community could use and enjoy.

This is why we love Poetry.

We want to be able to use Poetry for our developer workflows and apply the same philosophy of great developer experience, simplicity and productivity. Similar to https://github.com/sdispater/poetry/issues/241#issuecomment-438688395, our workflows typically include

poetry format
poetry lint
poetry build
poetry test
poetry coverage

We choose poetry amongst a dozen of other tools. We choose it to standardise across teams and to create common workflows. We choose it because we believe in the mission and the vision. We want Poetry to take over a bit more of our project workflows.

We can draw inspiration from the great success of npm scripts and the packaging and workflow advances that JavaScript community has been able to achieve in a relatively short period of time, or from decades of work that went into maintaining Makefile based toolchains, or other tools. But we :heart: Poetry.

[brycedrennan]

Poetry is a great tool. I'm a believer in "do one thing and do it well". I think asking for Poetry to become a task runner is asking for it to not stay focused on its core competency area: Python dependency management and packaging.

That's a hard enough problem to tackle without trying to also tackle related ones at the same time. Likewise, I don't think poetry should attempt to handle python version management, testing, linting, etc...

Pipenv supports custom scripts, but I think one of the issues with pipenv is that it tried to do too much.

Regardless, no biggie if something like poetry run gets added. It just shouldn't be conflated with setup.py entry_points or scripts. Nor would I want to add it just because people were unaware of Makefiles.

Agree to disagree about Javascript packaging being a model to follow 😄

[purificant]

No one is asking for a task runner, please do not get confused.

Poetry should and does handle versioning, may I refer you to https://github.com/sdispater/poetry#introduction for an example TOML file as well as https://github.com/sdispater/poetry#version and poetry help version. You must specify the python versions for which your package is compatible.

To avoid other sources of confusion, for pipenv related discussion, please check out https://github.com/sdispater/poetry#what-about-pipenv and the project's mission statement on https://github.com/pypa/pipenv

test and lint, etc. are all part of the developer workflow when working on a package, similar to existing poetry publish, poetry install, poetry update, poetry build. Likewise poetry run exists. If you don't like the CLI interface perhaps you should stick to writing Makefiles.

I am not entirely sure what point you are trying to make by pointing to a year old archived repo. Either way, it's hard to dispute that JavaScript community has been able to iterate very quickly on packaging and flesh out many ideas along the way. Python packaging community can benefit from their experience and learn from both successes and failures.

At the moment, the CLI interface for NPM is somewhat similar to Poetry's CLI, in that it creates good UX for developer workflows. The existing scripts can already be leveraged to achieve what are essentially dev-scripts for package workflow ( lint, test, coverage, etc. ) but this approach runs into several issues: it's not as clean as npm scripts, I think we can do better; it works reasonably well for codebases that do not need to be on PyPI or installed as a dependency, but for those scenarios where packages are installed using scripts results in essentially polluting global namespace.

There is some good work going on in https://github.com/sdispater/poetry/pull/591 that may get us there.

[brycedrennan]

@purificant

I'm not sure what you mean by task runner. I'm referring to a thing that runs arbitrary developer tasks. Several people are indeed requesting that in this thread. I think there are good reasons for wanting that (even though I don't think poetry should add that).

yes I'm aware that poetry manages packages and dependencies as they relate to python versions. It does not handle installation and management of python versions themselves. As I mentioned earlier, pyenv is a tool that does that. This is what I was referring to when I said poetry should not handle python version management.

I think we're all entitled to voice our opinions here with the shared goal of creating tools that make our lives easier. No need to suggest I "stick to writing Makefiles". We're all on the same team here.

It sounds like we have a similar understanding about what scripts in poetry are currently for: to install commands into the global namespace. We agree that as currently formulated it would be dangerous and inappropriate to use that feature for dev-scripts, because as you pointed out, it pollutes the global namespace.

To summarize:

• this issue (#241) is requesting feature parity with setuptools scripts
• setuptools scripts is intended to help python packages install useful command line tools for users. They exist in the global environment and are thus not appropriate for use with developer tasks.
• because npm has a feature called scripts that does something totally different (run developer tasks specific to a project), there was some discussion/confusion of that functionality in this issue.
• separately, somebody went ahead an implemented the dev scripts feature here. : https://github.com/sdispater/poetry/pull/591
• it would be a bad practice to try and fit developer tasks into the existing poetry scripts or even the version proposed here in #241. Until something like #591, is merged in, use something else... like Makefiles 😉

I suggest any further discussion of developer tasks happen in #591.

PS. Re javascript: I agree that my point about javascript was vague. I should have just not mentioned javascript so as to avoid a flame war. Agree to disagree. Happy to go into a big javascript debate in a different forum though. 😄

PPS. just for fun, one more makefile :smile:

[purificant]

I'm glad you now see with some of my earlier points, however, there are still a few things I could clarify.

I think I should re-iterate that if you like tool X, currently use tool X, perhaps for you it's a good idea to stick with tool X. This issue is not about tool X.

Since the first comment on the issue, the issue took a life of it's own as demonstrated by the number of upvotes on various comments. See https://github.com/sdispater/poetry/issues/241#issuecomment-398750736 https://github.com/sdispater/poetry/issues/241#issuecomment-412501335 https://github.com/sdispater/poetry/issues/241#issuecomment-431122114 https://github.com/sdispater/poetry/issues/241#issuecomment-431533126 https://github.com/sdispater/poetry/issues/241#issuecomment-431624301 https://github.com/sdispater/poetry/issues/241#issuecomment-438688395

for what this issue is now about.

[floer32]

I added a suggestion for an alternative to the #591 PR - agree with @brycedrennan that we can mostly move discussion of running 'tasks' to there. Just wanted to note this bit responding to OP

It would be nice if poetry supported the scripts feature of setuptools (note: this is not the same as entry points). [— @ojii]

If Poetry went with that suggestion / #940 , then it would be more similar to to that setuptools feature than "arbitrary inlined things to execute". Point being, in setuptools it explicitly only copies script files over, it avoids having people inline too much into that file.

[snowwm]

A new suggestion

Since #591 is closed and this issue is open, I assume the discussion should continue here (I may be wrong).

I completely agree with these words from @sdispater:

Poetry is a package and dependency manager, not a task manager. This feature is beyond the scope and the original purpose of Poetry and will likely never be integrated into it. I want to keep Poetry simple and intuitive so I carefully think before adding new features to the core codebase since the weight of maintaining it will fall on the shoulders of the maintainers.

However, what about adding a tiny piece of functionality that would be easy to maintain, yet provide the much sought-after concept of "single source of truth" (at least for task invocation, if not for definition).

First, consider a newly started project, whose author isn't old enough to know about Makefiles (I have nothing against Makefiles, but still). They would appreciate the possibility to just write:

[tool.poetry.tasks]
test = "pytest tests/"

Next, when they are ready to pick up a task runner, they just add a catch-all task redirecting all unrecognized commands to that external runner. Note, I do not suggest adding generic wildcards, just a single special case:

[tool.poetry.tasks]
"*" = "make" # invoke/doit/etc
test = "pytest tests/"

Now, the following happens (of course, tasks are run inside the venv): poetry task test -- --some-arg -> pytest tests/ --some-arg poetry task lint -- --another-arg -> make lint --another-arg External runner can be forced like this: poetry task -- test --some-arg -> make test --some-arg

The key point here is that when users ask for more complex functionality, we just point out that they should delegate it to other tools. But this switching stays transparent to most contributors of that project. Yes, my proposal is about an extra level of indirection. This way Poetry developers can put aside even the OS/shell compatibility concerns. The users should take care of it on their own, but for simple commands nothing is needed.

This proposal is orthogonal to #940 ("concise run"), which was also suggested as a means for running dev-scripts. I think, this design has one significant advantage: you can switch task runners without modifying commands -> transparently to most users.

And finally, I think this proposal could be implemented as a plugin at first, then merged into the main codebase, if successful. But I don't feel ready for starting this project.

[brycedrennan]

@snowwm I think your comment belongs in #591 or as a separate issue. This issue is for adding the scripts functionality as found in python-packaging - not for anything relating to a dev task runner.

[snowwm]

@brycedrennan I thought it has grown into a discussion about dev task runner, but you are right, that's not an excuse. So, should I move my comment there?

[peterdeme]

Guys, what's the situation on supporting scripts keyword? This one is a huge blocker for us.

sidenote: if Poetry builds a setup.py anyway why does it use different terminologies? eg.: scripts (Poetry) <-> entry_point (setup.py) [this_feature] (Poetry) <-> scripts (setup.py)

edit: whatever, I just added a PR, I'm hoping it can get merged!

[CathalMullan]

I've been using Taskipy for arbitrary script execution with Poetry. Has served me well so far.

[AndrewSpittlemeister]

I'm not trying to promote my own project, but this thread was the reason I created borca, so I thought I would share.

Features:

• Uses pyproject.toml to define tasks (kinda like Taskipy mentioned above, unfortunately that wasn't here last time I looked).
• Builds acyclic task graph
• Incremental builds with optional task state caching
• Inherently allows you to chain sub-projects together

[yajo]

I use https://www.pyinvoke.org/

I guess there wouldn't be a problem on poetry run invoke my-custom-invoke-task.

@pykong You want to read this thread :wink:

[hg-zt]

Well, I just want to package and install a shell script. Do we have a workaround now?

How could I make this work?

[tool.poetry.scripts]
# other entry points
my_shell_command = { path = "bin/my_shell_command.sh" }

[mattjw]

@Yajo

I use https://www.pyinvoke.org/

I guess there wouldn't be a problem on poetry run invoke my-custom-invoke-task.

I've been running with the same combo (poetry + invoke).

We used to use pipenv + invoke. And had pipenv's scripts as lightweight aliases into invoke tasks, with the exception of one or two commands that could have just become small invoke tasks anyway. Seemed sensible at first – reduce the amount you have to type to run a command. E.g., pipenv run test rather than pipenv run inv test. But actually for larger projects and teams it meant devs had two things to read and maintain. We've begun dropping scripts altogether and feel it's better.

So I've personally gone from "I want scripts in poetry" to "no need for scripts in poetry".

That said I can see why some projects (particularly smaller ones) might not want to go all-in on a runner like invoke, and would prefer simple lightweight functionality in poetry.

[djbrown]

A PR was closed referencing a plugin system https://github.com/python-poetry/poetry/pull/591#issuecomment-504762152 I use django and would like to use django commands and custom-management-commands directly via poetry wihtout writing poetry run python src/manage.py <command>.

[k4ml]

A PR was closed referencing a plugin system #591 (comment) I use django and would like to use django commands and custom-management-commands directly via poetry wihtout writing poetry run python src/manage.py <command>.

One problem is with django runserver command. When using autoreload (the default) it will try to rerun the command but will fail as scripts in poetry (if not build and install the package) is not a real executable or file. So django runserver will fail with:-

/home/kamal/.cache/pypoetry/virtualenvs/plum-QDTACV4j-py3.8/bin/python: can't find '__main__' module in 'yourscript'

This is because django will see the full command to execute as:-

['/home/kamal/.cache/pypoetry/virtualenvs/plum-QDTACV4j-py3.8/bin/python', 'yourscript', 'manage', 'runserver']

But if the package is installed, then django will see the command as:-

['/home/kamal/.cache/pypoetry/virtualenvs/plum-QDTACV4j-py3.8/bin/python', '/home/kamal/.cache/pypoetry/virtualenvs/plum-QDTACV4j-py3.8/bin/yourscript', 'manage', 'runserver']

which is correct.

[cpvandehey]

One workaround that I've put in my codebases is a copy command within a dockerfile:

RUN cp myfile.py ${VIRTUAL_ENV}/bin/

This is an ugly workaround for now until Poetry can figure out how to replicate the scripts sections from setup.py. This is such a reasonable request; I can't believe we are still struggling with a copy paste feature.

[peterdeme]

Closed my previous PR, opened a new one for this feature: https://github.com/python-poetry/poetry-core/pull/40

[roipoussiere]

I just want to share my workaround for using development tools, while waiting for the plugin system and the tasks plugin to be released.

I do not fill the [tool.poetry.scripts] for development purpose, but instead I write a builder.py script that accepts several commands such as lint, test, etc.:

def lint():
    # ...
    pass

def test():
    # ...
    pass

if __name__ == '__main__':
    commands = {
        'lint': lint,
        'test': test
    }

    if len(sys.argv) == 1 or sys.argv[1] not in commands.keys():
        print('Usage: make %s' % '|'.join(commands.keys()))
    else:
        commands[sys.argv[1]]()

I also create this simple Makefile:

%:
    poetry run python ./builder.py $@

Then I use it with make test, make lint, etc.

The command passed to make is passed to the builder script, which is always executed in the Poetry virtual environment.

And you don't really have to learn how to write Makefiles, because here the Makefile content doesn't change, you just have to update your builder some times. ;)

edit:

I finally prefer to list all the targets in the Makefile in order to use Make auto-completion:

lint:
    poetry run python ./builder.py lint

test:
    poetry run python ./builder.py test

[nat-n]

Since this PR probably still gets some google traffic I'd like to share a solution I'm working on:

Poe the Poet a task runner that works well with poetry.

Basic usage is quite similar to proposal in this PR, but it also supports defining tasks using shell syntax or as references to python functions similar to scripts.

Besides lightweight configuration contained in the pyproject.toml, one key advantage over make is that you can easily pass extra arguments to tasks without extra configuration.

Example configuration with different task types:

[tool.poe.tasks]
test       = "pytest --cov=poethepoet"                                # simple command based task
mksandwich = { script = "my_package.sandwich:build" }                 # python script based task
tunnel     = { shell = "ssh -N -L 0.0.0.0:8080:$PROD:8080 $PROD &" }  # shell script based task

Example invocation:

# extra argument is passed to the underlying pytest command
poe test -v   

# If poethepoet is added as a dev-dependency
poetry run poe tunnel

The intention is to eventually have comparable functionality to make wrt task composition, conditional execution, and command line completion.

EDIT: as of v0.12.0 it can also be used as a poetry plugin