Open ruben-arts opened 1 month ago
Hofer-Julian
commented
1 month ago One thought: pixi env doesn't make clear that we talk about global environments here. But adding a global prefix will clash with pixi global install. Naming that feature will not be obvious.
maresb
commented
1 month ago My initial thought is to not reinvent the wheel.
From my perspective the useful part would be
pixi create-external-env -p destination-prefix -e pixi-environment-nameand then let the user use micromamba from that point.
ruben-arts
commented
1 month ago @maresb from the single command I'm not getting enough information to get the gist of your proposal.
We're not looking to reinvent the wheel here, it's more of adding another wheel to the oiled machine that is pixi :wink:
Having the conda like workflow within the pixi tool is something a lot of people have been asking for. Especially without having to install any other tools. So we're openly brainstorming what that would look like, without having a 1-on-1 implementation of conda's cli.
It sounds that what you are looking for might be pixi project export https://github.com/prefix-dev/pixi/pull/1427 so you can switch between the tools you like. That is a different story from this Issue.
maresb
commented
1 month ago EDIT2: I don't like what I wrote, and I think @synapticarbors is saying something very similar and articulates it much better, completely and simply.
ethanc8
commented
1 month ago As a quick minimal solution, could we expose the configuration option detached-evironments as an environment variable or CLI flag? Then you can create environments using PIXI_DETACHED_ENVIRONMENTS=1 pixi install --manifest-path /path/to/pixi.toml and enter the environment using PIXI_DETACHED_ENVIRONMENTS=1 pixi shell --manifest-path /path/to/pixi.toml
synapticarbors
commented
1 month ago @ruben-arts -- Thank you for introducing this. I'm definitely one of those users that would benefit from an alternative workflow. In thinking about what you've proposed, you said this is not going to be "a moved pixi project". I think this may result in a missed opportunity and result in something more limited than "pixi with a conda workflow". What I would want as a user of both pixi and conda is something that is just pixi with a global project/env registry, so the workflow would look something like:
# Create and register a new project in ~/.pixi/envs or some other configurable location
# (i.e `conda create -n test-proj` but no initial dependencies)
pixi global-env init -n test-proj
# List registered global projects (i.e `conda info --envs`). Perhaps also list the environments of each project
pixi global-env list
# "activate" the project. Optionally specify environments through `-e` flag
pixi global-env shell -n test-project
# delete global project
pixi global-env remove -n test-projectIf pixi global-env shell -n test-project binds pixi to that project/env by making the manifest-path persistent within the shell, then within that shell, I should be able to do all of the things I would normally do with pixi, but without having to be in the project directory or specify --manifest-path.
My ideal here would be to have a very lightweight global-env (or global-proj) subcommand, and then have the user drop into a pixi shell, and have all of the features of pixi. I'm not sure if there are major technical issues to making it work like this, but my hope would be that this would way of doing things that wouldn't undermine a bunch of the features that makes pixi great, allowing users to have the best of both worlds (if they choose).
ruben-arts
commented
1 month ago @synapticarbors Thanks for you comment!
It sounds like you would just like to add a manifest-path to a map of environment names to manifest paths.
This might not be a completely different subcommand but more of an extra feature. e.g. by default we register the name in a file that keeps track of all the projects. And on duplicate we spawn a interactive request or simple overwrite the name to the latest modified project with that name.
Example of that map:
pixi = "~/development/pixi"
rattler-build = "~/development/rattler-build"Then adding a --global or --detatch or similar flag to the init command. To move it under the ~/.pixi/envs directory.
e.g.:
> pixi init my_env -g
Created project in ~/.pixi/global_projects/my_envResulting in the mapping file including:
pixi = "~/development/pixi"
rattler-build = "~/development/rattler-build"
my_env = "~/.pixi/global_projects/my_env"allowing for
> pixi shell -n my_env
(my_env) > pixi add python
Added python >=3.12.3,<4 to my_env
(my_env) > What I'm affraid of, is that pixi projects are designed with much more features than a conda environment. Think tasks, multi-environment setups, system-requirements. All these features don't fit into the conda workflow and might spook or influence the expected behavior.
That all said, this could be a first step and possibly enough for most users.
ethanc8
commented
1 month ago It sounds like you would just like to add a manifest-path to a map of environment names to manifest paths.
I would also like this.
All these features don't fit into the conda workflow and might spook or influence the expected behavior.
I don't really think that's an issue -- the point of this is to support workflows that conda supported and pixi hasn't really supported while bringing pixi's unique features to those kinds of workflows. Anyone who wants extremely conda-like behavior can just use conda, mamba, or micromamba, or a new minimalistic rattler client could be built for them.
synapticarbors
commented
1 month ago @ruben-arts I'm sure you've talked to more people about this, but just based on various discussions here and on discord, I think what most people want when they say a "conda-like workflow", is just pixi with an optional global registry of environments like conda. What you were suggesting with:
# create project in global registry
$ pixi init my_env -g
# activate env in global registry (with optional -e flag)
$ pixi shell -n my_env would satisfy this as long it things worked as expected if you were to use further pixi commands (e.g. pixi run, pixi add, pixi list, etc) in that shell and have pixi behave like it was in that global project's directory. I think if you keep it simple like this, you minimize the risk of bifurcating the pixi user base into people getting the full pixi experience based on a project-based workflow vs a greatly limited conda-based one. I don't think there is a reason to try to directly map commands onto a conda-like interface. I think users will adapt if they have this alternative mechanism.
ethanc8
commented
1 month ago I think another way would be to allow simply naming the manifest files. For example, when I run pixi shell --manifest-file=my-project/pixi.toml -n my-project, then in the future I'd be able to run pixi shell -n my-project, with pixi updating the environment whenever I change the manifest file at my-project/pixi.toml. This way, it'd be easy to edit the manifest for a project while still using that environment in related projects.
matrss
commented
3 weeks ago While I understand the desire to have a familiar workflow, I think it is an inherently broken one that shouldn't be supported. Having environments that can be enabled anywhere encourages re-using these environments between unrelated projects, which will inevitably lead to issues where changes done for one project will break another. This in turn means that there will be no useful documented computational environment for either project, breaking one of the main advantages of pixi IMO.
To avoid this issue you need one environment per project, at which point you can just use normal pixi projects.
What I do see some value in is having a singular global environment that is always available, to install packages in that are always needed and not tied to a single project (e.g. git, git-annex and DataLad in my case). This already exists with pixi global install, although it doesn't work with python packages since each package installed with it is technically in a separate environment so python doesn't find the modules. But I also cannot think of a python module that I would want to install in this kind of way, anyway.
ethanc8
commented
3 weeks ago Having environments that can be enabled anywhere encourages re-using these environments between unrelated projects, which will inevitably lead to issues where changes done for one project will break another.
It's quite often that you would have multiple environments that are usable for a project or multiple related projects in different directories. Also, you might be working on a project and one of its dependencies at the same time, in which case it'd be useful to share an environment between them. There are many more examples of usecases where you would want a project to have the same environment. Lastly, small "playgrounds" where you just need to write some code and test it against some dependencies, to understand how the dependencies work, are also useful, and it's not useful to create a new environment just to test a single dependency, especially if you're planning to use the information about your dependencies' behavior or API in an existing project.
apcamargo
commented
3 weeks ago I'll share my thoughts as someone who has been using Conda for ~6 years in computational science (specifically, bioinformatics). It seems to me that the project/directory-centric workflow works great for software development, where I agree that reusing the same environment across multiple projects is beneficial. However, in bioinformatics, we typically don't work that way (though I'm using "we" loosely here, as this is based on personal experience rather than broad consensus). We often create environments to install the software required to run specific pipelines and to avoid conflicts between different pipelines or even different steps within the same pipeline. Forcing users to always run pipeline X in the same directory might be too restrictive since pipeline X could be executed on very different datasets across distinct scientific projects.
I also wonder how something like Snakemake would work within Pixi's current paradigm, considering it creates Conda environments on demand for different steps of a workflow, all within a single directory as the workflow runs (or at least that's my understanding of it).
It would be great to hear from others in the BioConda community, as they might have different perspectives (and might even disagree with me).
matrss
commented
3 weeks ago It's quite often that you would have multiple environments that are usable for a project
Pixi supports multiple environments per project already.
It's quite often that you would have multiple environments that are usable for [...] multiple related projects in different directories.
Again, I think that this is never a good idea because you will never be able to name the exact environment you have used in your project without making it exclusive to the project.
All cases that I can think of in which someone might want this fall into one of two categories:
Also, you might be working on a project and one of its dependencies at the same time, in which case it'd be useful to share an environment between them.
I don't think so. If one project depends on the other in the sense of it needs the other installed as a software, then it should just be declared as a dependency (Granted, I don't know if it is possible with pixi to install a dependency from another local project. Probably only as PyPI dependencies, currently.). If the other project is otherwise free-standing, then I would want it to have its own environment. If both depend on each other then they might as well be the same project anyway.
Lastly, small "playgrounds" where you just need to write some code and test it against some dependencies
That's what I have a ~/Playground directory for, in which I can just create random projects as needed. Again, grouped together for easy cleanup.
[...] and it's not useful to create a new environment just to test a single dependency, especially if you're planning to use the information about your dependencies' behavior or API in an existing project.
If you plan to use this testing in an existing project, then just do it in that project. This is what branches are for, no need to set up a completely separate test project/environment.
We often create environments to install the software required to run specific pipelines and to avoid conflicts between different pipelines or even different steps within the same pipeline. Forcing users to always run pipeline X in the same directory might be too restrictive since pipeline X could be executed on very different datasets across distinct scientific projects.
How do you tie it all together when your pipeline has produced a result and you want to reference it e.g. in a publication? You would have to reference the exact version of the pipeline, the pipelines environment(s), and the data. In my opinion this is only really feasible when bringing all three together into one project, e.g. with DataLad to store pipeline and data, and pixi (or nix, or guix, or another similar tool) to describe the environment.
Users shouldn't be forced to always run pipeline X in the same directory. They need to create a new project for their intended data processing, install the pipeline and the necessary dependencies into it, and then run the pipeline on their data.
I would see some value in templating a pixi project with a common set of dependencies for this purpose, but in the end they must be distinct projects so that updating pipeline X and its environment does not break the provenance of the results processed from some data using a previous version of that pipeline.
This is just my opinion though, from the somewhat idealistic viewpoint of making (scientific) results traceable to their origin and giving them enough accompanying metadata to make it even remotely feasible to understand how they were conceived and what issues there might have been (or not) in the process.
synapticarbors
commented
3 weeks ago @matrss -- While I appreciate that you have a clear idea of what a scientific workflow should look like, it's mildly off putting to say that other workflows are broken or wrong. I think the goal here is to make the transition to pixi easier for the large community of conda users who have a particular workflow that differs from what pixi currently offers. I'm guessing a lot of people try pixi, find it doesn't work with their workflow and then return to just using conda/mamba. It's better to give them an option, and then when they stick around, maybe they will adopt the project-based workflow.
matrss
commented
3 weeks ago I didn't say that other workflows are wrong, I said that I think the workflow induced by conda/mamba environments is fundamentally flawed and shouldn't be encouraged by supporting it, and I also explained why I think that way.
Yes, sure, it might make the transition to pixi a bit simpler for some, but is that worth diluting the major benefit of using pixi over conda/mamba, instead of communicating that benefit and encouraging taking advantage of it?
Yes, giving an option might get people to use pixi and adopt a project-based workflow later on, but it also makes it simpler to not re-consider a flawed workflow and stick with the familiar, instead of adopting something much better.
ethanc8
commented
3 weeks ago There are still many reasons why one would not want to use the workflow that you claim is "much better". Meanwhile, pixi has many advantages over conda/mamba other than its opinionated workflow, and for many users including myself, those other advantages are much more important than its opinionated workflow. For example, once pixi build is available, that would become one of pixi's largest advantages. In addition, pixi supports PyPI/wheel packages much better than conda/mamba, has a pixi.toml which is better than environment.yml, shares some of its codebase with rattler-build (which is useful for rattler-build contributors and users), and has a nicer CLI. I don't see why we should limit all of these advantages to use the opinionated workflow when there is no technical reason we couldn't support all of these things with the traditional conda/mamba global environments workflow.
apcamargo
commented
3 weeks ago How do you tie it all together when your pipeline has produced a result and you want to reference it e.g. in a publication? You would have to reference the exact version of the pipeline, the pipelines environment(s), and the data. In my opinion this is only really feasible when bringing all three together into one project, e.g. with DataLad to store pipeline and data, and pixi (or nix, or guix, or another similar tool) to describe the environment.
If you use Snakemake, for instance, the environment files for each step in the pipeline will be there. Even if you don't, it's not that hard to generate files describing your exact environment.
I don't disagree with you that pixi's approach has benefits. My point is that these are almost always discussed in the context of development environments. Maybe allowing Conda-like environment management can have benefits in other contexts, such as scientific research. I don't have strong opinions, just want to bring a different perspective (that might not even align with peers').
ruben-arts
commented
3 weeks ago Hi all, thanks for all your feedback.
This is a great discussion but I would like to make it a little more concrete. Could you share your use-case in a user story format.
e.g. “As a [persona], I [want to], [so that].”
This would help us greatly, thanks in advance.
matrss
commented
3 weeks ago There are still many reasons why one would not want to use the workflow that you claim is "much better".
Maybe, but I also didn't say that pixi's project-based workflow is always better than everything else. I said that every instance in which someone might think that they want to use conda-style environments, they would be better served by using pixi projects. So, there might be reasons why one would not want to use what I called "much better", but I don't see any situation in which conda-style environments would then be a better pick than pixi's project-based workflow. If you can name a situation that doesn't fit into one of the two categories I mentioned then please do so, I genuinely can't think of any.
For example, once
pixi buildis available, that would become one ofpixi's largest advantages.
I am not sure how you would want to take advantage of pixi build without associating the build environment to the thing to be build.
In addition,
pixisupports PyPI/wheel packages much better thanconda/mamba
True enough, yes.
has a
pixi.tomlwhich is better thanenvironment.yml
Yes, but IMO pixi.toml is much better due to the underlying difference in workflow. When you try to use it to simply define a set of installed packages, then it becomes an environment.yml in toml format, without much gain.
shares some of its codebase with
rattler-build(which is useful forrattler-buildcontributors and users)
Fair.
and has a nicer CLI
Implementing conda-style environments in pixi will necessarily weaken this point, since the CLI to interact with them must be similar to conda's CLI, otherwise you wouldn't get the benefit of familiarity.
If you use Snakemake, for instance, the environment files for each step in the pipeline will be there.
OK so I don't have much experience with snakemake, but my understanding of snakemake is that the environment files are part of a snakemake pipeline definition, which resides in a directory. So they are part of the project that defines the pipeline, which in turn will be referenced in a specific version to apply the pipeline to something (either by directly running it in that project, or including it in a different project's snakemake workflow).
I don't see why the snakemake pipeline couldn't use a pixi project to define these environments (utilizing pixi's multi environment feature), other than that snakemake does not yet support pixi in that way. But it would need to be extended to support pixi's hypothetical conda-style environments as well, so that is not a downside, just some necessary integration work.
What it does with environment files is already just a more brittle version of using a pixi project to define these environments. IMO it is not necessary to weaken pixi's benefits to support this use case, it can be implemented with what pixi has already.
Maybe allowing Conda-like environment management can have benefits in other contexts, such as scientific research.
That's the thing, I don't think it can have benefits in any context and instead think that conda-like environments are an anti-pattern that encourage bad practices, especially in scientific research. I would be happy to be proven wrong though.
e.g. “As a [persona], I [want to], [so that].”
As someone involved in research data management and wanting to make research more traceable and reproducible (improving mainly the last two aspects of the "FAIR" principles), I want to encourage people to publish artifacts with their research that fully encapsulate the process that they followed from raw data or even data acquisition to the results they wrote about, so that anyone is able to go from e.g. a plot in a paper back through the exact processes used to generate it (this includes the entire computational environment, all pipelines, etc.) and arrive at the data from which one started. I see pixi's currently implemented project-based workflow as one potential piece of this puzzle, to clearly define the computational environment(s) used. Conda-style environments are fundamentally not compatible with this vision, unless they are used in a way that simply emulates a project-based workflow. I have been recommending pixi over conda/mamba to colleagues for this reason and if pixi implemented support for conda-style environments this would make it harder for me, since I would always have to add the disclaimer "but don't use these conda-style environments, they will inevitably break reproducibility if you don't limit yourself to a single project per environment, at which point you can just use the easier to work with project-based workflow".
dhirschfeld
commented
2 weeks ago I think this exposes a fundamental schism between software engineers and scientific development which IME is exploratory, REPL-driven development, NOT project based development.
It is simultaneously true that:
As a quant analyst who has moved more into development I tend to use more (pixi) project based development in an IDE such as PyCharm or VSCode, but when I want to test things out I use Jupyter Notebooks in my kitchen-sink environment.
There are different stages of development and when you just want to test some Python code out against your latest models you don't want the overhead associated with creating an arbitrary "project" just to run some code. If I want to run arbitrary code against arbitrary dependencies I have a dev environment I can do that in with zero overhead. I have dev-310 and dev-312 environments for testing arbitrary snippits against anything I might want. Each of those environments has all the tools installed to pull data from any system I might need and has all of our latest code installed (in editable mode).
I only create "projects" when I want to actually develop/deploy something and make it usable by others. This is a different phase of development from the exploratory analysis phase. It's not a phase most scientists are familiar with so it's important to make it easy to transition to a more robust/reproducible project based workflow, and I think pixi could be an incredibly powerful tool for that... but only if there is some way to support both workflows (whilst making the project workflow the default).
I'm (one of) the biggest pixi fans around but working in data science I didn't make the switch until I could at least simulate a conda env with pixi shell --manifest-path. Forcing users to create a "project" and install the myriad dependencies which may be required to test the latest and greatest code adds so much overhead to the exploratory development workflow that it makes it ~infeasible as compared to activating an environment with all the necessary dependencies already installed.
cdeil
commented
2 weeks ago I've been using conda for 10 years and like it. Thank you all for making the conda package ecosystem better and better and now creating pixi!
I would love to adopt pixi as a drop-in replacement and just use pixi in the next years instead of a pixi / conda / pip combination. Similar to what some others said here, I do have the use case to have personal "dev" and "play" conda envs and I also sometimes use a conda "base" env just for generic Python scripts or one-offs where I don't create a project and manifest files.
So I'm +1 to extend pixi to allow creating / managing / deleting "global envs" just like conda does.
Why not keep using conda? Well I will for now. But pixi would give me (a) better speed and (b) easier bootstrapping and (c) consistency with the pixi workflow and specs I'm happy to adopt for projects where I an convince collaborators - usually long-time conda users like me.
Made one concrete related suggestion here: #1877
maresb
commented
1 week ago As one of the developers of PyMC, we encourage users to set up Conda environments to ensure that beginners can easily set up the required C compilers and BLAS. We end up dealing with a large number of beginners asking for help with broken Conda environments. I would like to add pixi to our installation instructions since I believe it's a major step up from other tools and would be much simpler for everyone in the long-term. (I believe that pixi's isolated project-based config would fix most causes of broken environments.) However, many of these beginners are copy-pasting instructions without understanding what a Conda environment actually is. Some of these people will encounter problems and then run any command they can find in an attempt to "just fix it" without understanding what's broken. While pixi env in capable hands would undoubtedly be very useful, I'm concerned that pixi env could become a source of confusion for such beginners.
In order to add pixi to the PyMC installation instructions I must convince skeptical comaintainers that the support and maintenance efforts are worthwhile. If pixi adds a pixi env subcommand then this may increase our support burden. Maintainers are comfortable with conda and to a lesser extent mamba and micromamba. Already the subtle differences between conda and micromamba lead to confusion, so a pixi env subcommand that only partially attempts to reimplement the conda interface may lead to difficulties and make it harder for me to argue to them that pixi is complete and stable enough for inclusion. From this perspective I'd be a lot more comfortable with this proposal if we picked out a few essential environment-related features and gave them a distinctive interface that can't be confused with the conda interface.
Thanks everyone for the great discussion. I appreciate the focus on the user story approach.
ruben-arts
commented
1 week ago After going over this issue with the core team we've got a specific proposal based on the comment by @synapticarbors
Create new project and register it immediately
pixi init project_a --register
# or using the current folder:
pixi init --registerRegister an existing project with an optional name that you could give the project otherwise default to the name in the manifest.
cd project_b
pixi project register
pixi project register optional_nameThis would add the following field to your configuration.
registered-projects = { project_a = "/path/to/project_a", project_b = "/path/to/project_b", optional_name = "/path/to/project_b" }Then you can run the following commands
pixi run --name/-n project_a start
pixi shell --name/-n project_b
pixi shell-hook --name/-n optional_nameBasically adding the --name to all the commands that also use the --manifest-path.
I hope this fits the most use-cases while staying true to the pixi workflow.
dhirschfeld
commented
1 week ago It sounds great to me & I'm keen to test it out!
akhmerov
commented
1 week ago I am not sure whether this question belongs to the repo, but I would like to ask how this feature would interact with pixi jupyter kernel?
synapticarbors
commented
1 week ago Awesome! Thank you to the pixi team for taking the time to consider this.
dhirschfeld
commented
1 week ago I am not sure whether this question belongs to the repo, but I would like to ask how this feature would interact with pixi jupyter kernel?
I just write my own kernelspecs. for pixi I imagine it would look like:
{
"argv": [
"pixi", "run", "-n", "my-env",
"python", "-m", "ipykernel_launcher", "-f", "{connection_file}"
],
"display_name": "Python (my-env)",
"language": "python",
"env" : {
"A_VARIABLE": "a_value"
},
"metadata": {
"debugger": true
}
}
EDIT: This initial proposal is outdated because its already clear this is not what the community wants. Please follow the thread as there is no clear conclusion yet!.
One of the most asked features I hear constantly is "conda drop in replacement". Note that this would not be a move against conda but a request I would like to fill in for the users.
This issue describes a feature that gives a more conda-like feel to the pixi environments.
Workflow example
Create an environment
Activation
Add packages
Remove packages
In the background
.pixi/envspixi.tomlfile to the.pixi/envs/test-env/conda-meta/pixi.lockfile after solve to the.pixi/envs/test-env/conda-meta/What it should be
condais replaced withpixi envuvlike pixi projects do, instead of callingpipfor thepypidependencies.What it is not going to be
condas CLIconfig.tomland not conda's.condarc. Mixing those two configuration files is going to be a mess. We're much more likely to add a converter between these files then to support both in one tool.tasks,features,system-requirements, etc.Ps. this is a start of the design, tracking the interest and a place to discuss design. After the design is finalized we can start building and other contributors are free to help.