prefix-dev / pixi

Package management made easy
https://pixi.sh
BSD 3-Clause "New" or "Revised" License
2.19k stars 132 forks source link

Binding tasks to environments directly #1519

Open hugokerstens opened 2 weeks ago

hugokerstens commented 2 weeks ago

Problem description

From the docs I understand that you can configure a task to be bound to a specific feature. As far as I understand, binding a task to an environment is not possible. However, when running an (unbounded) task from the CLI it's the other way around: you of course have to specify/select the environment to run the task in, not the feature. This behavior is unintuitive to me.

Is it by design that tasks are not able to be bound to environments? Or is this something that is not implemented yet?

Example use case

Consider the following use case, where a task requires dependencies from multiple features.

I have a python package that includes a documentation with several scripts to generate some example plots using matplotlib (which is not a dependency of the python package). Locally, I use jupyter to develop the scripts. In CI, I use mkdocs to execute the scripts and build the documentation. My pixi.toml contains the following:

[feature.analysis.dependencies]
matplotlib-base = ">=3.8"

[feature.local.dependencies]
jupyter = ">=1.0"

[feature.docs.dependencies]
mkdocs = ">=1.6"

[environments]
local = { features = ["local", "analysis"] }
docs = { features = ["docs", "analysis"] }

[tasks]
# I want to bind this to the environment docs, not the feature docs,
# as the command requires both the analysis and docs feature.
build_docs = "mkdocs build"

The task build_docs belongs to the docs environment, but there's no way to define it that way. Binding it to the feature docs is not correct, as the command requires dependencies of multiple features.

I am interested to hear your thoughts on this!

vigneshmanick commented 1 week ago

if tasks that depend on each other are defined, then the plots from analysis will be executed before the mkdocs command is run.

In the example below based on your config, i have tried to create plots from the analysis feature which is then executed before the build_docs command is run.

Output

> pixi run -e docs build_docs
✨ Pixi task (plots in docs): python data/plot.py

✨ Pixi task (build_docs in docs): mkdocs build

Example

[dependencies]
python = ">=3.12.4,<3.13"

[feature.analysis.dependencies]
matplotlib-base = ">=3.8"

[feature.analysis.tasks]
plots = "python data/plot.py"

[feature.local.dependencies]
jupyter = ">=1.0"

[feature.docs.dependencies]
mkdocs = ">=1.6"

[feature.docs.tasks]
build_docs = {cmd="mkdocs build", depends-on=["plots"]}

[environments]
local = { features = ["local", "analysis"] , solve-group="default"}
docs = { features = ["docs", "analysis"] , solve-group="default"}
hugokerstens commented 1 week ago

Thanks for your reply. This approach is only feasible when the commands can be separated, but this is not always the case. In my example, mkdocs build executes the scripts internally.

My question can be summarized as: how do you handle when a single task requires dependencies from different features?

I think of it this way. When a feature-bound task is run, pixi will find an environment that includes that feature and execute the task in there. Why not allow to bind to a specific environment directly?

akhmerov commented 4 hours ago

I have another case that doesn't seem to quite map on the set of concepts implemented in pixi right now.

I have a package with basic and optional dependencies. These seem to belong to different features, and packaged used for testing into a yet another feature. I would then like to make test_comprehensive task which requires 3 features: basic_dependencies, optional_dependencies, and testing.

The question I fail to answer is: within which feature should I define test_comprehensive? It seems to be equally wrong to put it in any of the 3 features.