Allow `run`ing multiple targets from CLI or aliases

[huonw]

Is your feature request related to a problem? Please describe.

pants run ... allows running various types of scripts/tools/..., which is great. Unfortunately, one can only run one target at a time, e.g. various incantations like pants run foo.py bar.py or pants run foo.py run bar.py give TooManyTargetsException: Therungoal only works with one valid target, but was given multiple valid targets.

This restricts the ability to use pants as a orchestrator, e.g. I might want to do pants run prepare_database.py run web_server.py to first prepare the database that's used by a web server.

Currently, users are forced to use external orchestrators, e.g.:

• npm
• make
• shell scripts: https://github.com/komprenilo/liga/tree/89f925fbd2db34d30433abd5f6f5802619beb7f5/bin

This leads to a bunch of adhoc error checking (if it exists...), hacks around concurrency, more requirements what's installed on each user's system, and just general frustration of having to interact with those external system.

Describe the solution you'd like

Proposed requirements:

1. Allow running multiple commands in succession, including via aliases. (Some equivalent of pants run prepare_database.py && pants run web_server.py)
2. Allow passing arguments to those commands (Some equivalent of pants run prepare_database.py -- foo --bar && pants run web_server.py -- baz)
3. Allow running a command in addition to other goals (Some equivalent of pants run some_adhoc_formatting.py && pants fmt ::)
4. Potentially: allow running commands in concurrently (Some equivalent of pants run foo.py & pants run bar.py)

For example pants prep-db web or pants do-it-all would first run prepare_database.py and then run web_server.py, equivalent to pants run prepare_database.py && pants run web_server.py.

[cli.alias]
prep-db = "run prepare_database.py"
web = "run web_server.py"
do-it-all = "prep-db web"

Describe alternatives you've considered

An alternative would be being able to express this adhoc relationship via explicit dependencies "run prepare_database.py for its side effects before running web_server.py" (somewhat along the lines of https://github.com/pantsbuild/pants/issues/16362 or https://github.com/pantsbuild/pants/issues/17860).

python_source(source="./prepare_database.py")

adhoc_tool(
    name="run-prepare-database",
    runnable="./prepare_database.py",
    cache=False, # NEW: indicates side effects
)

python_source(
    source="./web_server.py", 
    dependencies_for_run_goal=[":run-prepare-database"] # NEW
)

But: being able to run multiple things adhoc in addition to explicit dependencies seems useful. As in, both multiple-run and better modelling of side-effect/preparation processes seems good.

(This alternative also likely wants https://github.com/pantsbuild/pants/issues/12794, because running run-prepare-database only makes sense when actually running web_server.py: packaging web_server.py into a pex_binary doesn't need to run that command.)

Additional context

Somewhat related issues:

• https://github.com/pantsbuild/pants/issues/10542
• https://github.com/pantsbuild/pants/issues/17729 (supporting run like this would reduce the need for this one, because adhoc tools could be run as part of a "goal" by just creating an alias that includes running the adhoc tool)

[jsirois]

N.B.: The driver here is -- .... The support for that is what has historically led to the simple answer of only allowing running 1 thing; then the target for the pass through args is unambiguous. Clearly that feature need not block, its just the historical impetus for the current situation.

[jsirois]

It might be good to explain why having Pants be the orchestrator instead of the shell is considered a good thing here.

Notably:

Allow running multiple commands in succession (bonus points: also allow running multiple concurrently), including via aliases.

Is clearly covered by (in bash), use of &, wait and &&. I maybe could guess why someone may not like using their shell for orchestration, but it really seems good to button that up just to ensure this would be a feature that pulls its weight or is necessary in some way.

[kaos]

Perhaps at least remotely related to this, I've recently started to ponder on a workflow backend for pants.. that is, targets for declaring what to do and in what order, with branching (based on outcomes etc) and joins.

This for one could address the common questions about pre-test and post-test setup/teardown--you'd simply setup a test workflow with pre and post steps surrounding the main test goal.