Open vigneshmanick opened 4 months ago
I think this would be a great enhancement. I've heard more feature requests like this. Simply to improve the deployment question.
Please get this upvoted! :+1:
Yep, this is a very important end state of a .conda
-related workflow, and is probably a hard blocker for a number of projects adopting pixi
.
Of note, this can probably already be done with the battle-hardened constructor
itself as e.g. [task.installer]
by pointing construct.yaml#environment
at an already-installed .pixi/envs/{env-name}
. I haven't verified this, but if not, it's probably a bug in pixi
.
Another intermediate point would be to be able to generate the @EXPLICIT
file for a given env, and use that as an input to constructor
. Again, this probably would already work with e.g. ((micro)?mamb|cond)a
to generate the well-known, if flawed, file.
To improve on the state of the art, an only-works-with-pixi
installer would need to do one or more of the following, without regressing on any of the others:
pillow
on windows)pypi
dependencies transparently (still an advantage of conda-pack
)SOURCE_DATE_EPOCH
).msi
conda
-like executable, and usually a python
)We developed pixi-pack which solves some of these use cases. It covers some (imo the most important ones) use cases mentioned here but not all. Feel free to check it out and see whether it covers your needs.
I personally think that this feature should belong into a different tool (like pixi-pack or if you think it should go in a different direction feel free to open issues there/create a fork) to not make pixi's CLI bloated. Also with pixi-pack's existence I don't feel for the need to reinvent the wheel here. WDYT?
With that being said, is this issue still relevant or should we close it? I could also write some documentation for https://pixi.sh that mentions how pixi-pack can be used.
Documentation is always welcome @pavelzw :)
Yeah, pixi-pack
looks useful in many scenarios... but a user still has to get pixi-pack
and the tarball.
Once a release lands with #1873, I'm planning to look more closely at the pixi.toml -> construct.yaml -> my-installer.(exe|sh|dmg)
toolchain. This should end up being a very small amount of plumbing, so could be a good writeup.
Not having a rattler
-based constructor wouldn't be as efficient as if pixi
, rattler-build
, and not-constructor
could share the same caches and hot paths for solving, downloading, unpacking, etc.... but heck, constructor
supports micromamba
, and maybe could be taught to speak pixi
.
One thing I haven't really looked into fully is the rust
-based tauri. This is compelling as .msi
is really the most adoptable output in a managed windows situation. The "native" web view piece on Linux (some gtk thing) wouldn't be my cup of science visualization tea, but would be totally sufficient for config, etc.
An interesting intersection is this example... this downloads a constructor
installer at run time, but presumably it could be made fully offline-capable as well. Or include pixi-pack
.
So yeah: I think continuously tested examples that end up in narrative docs will end up being the right call, as the matrix of use cases gets enormous at that end of the stack. For installers, getting to parallel, restartable execution, e.g. pixi run -j8 build-installer
would be the biggest boon.
but a user still has to get pixi-pack and the tarball
Yeah, pixi-pack looks useful in many scenarios... but a user still has to get pixi-pack and the tarball. +1
This issue is still relevant, my use case for constructor is a completely airgapped machine where neither conda nor other tools (pixi-pack) exist.
Just copying an environment elsewhere is, imo. not an ideal delivery mechanism since what is required is an end user usable artifact, not recreate a developers machine elsewhere.
In my case the installer that constructor produces
root:.
├───bin
└───start # this script sets the necessary variables/scripts required for activating and calling the environement
├───python # this is where the constructor artifact is deployed
└───envs
└───etc..
├───OtherData
└───Examples
I still belive that pack
could be part of a pixi project definition. There is pixi add
, pixi build
is on the way and deploy/pack
will complete the entire toolchain nicely.
my use case for constructor is a completely airgapped machine where neither conda nor other tools (pixi-pack) exist
pixi-pack is a statically linked executable, you can just copy it together with environment.tar
over to the air-gapped machine and it will work; no need to install anything
Just fyi this was an experiment to make a self-executable mamba constructor like thing https://github.com/mamba-org/monstructor
I think something similar could work for pixi pack where it embeds itself in front of the compressed data :)
my use case for constructor is a completely airgapped machine where neither conda nor other tools (pixi-pack) exist
pixi-pack is a statically linked executable, you can just copy it together with
environment.tar
over to the air-gapped machine and it will work; no need to install anything
nodejs
can be installed via conda but the npm dependencies have to be installed manually, will these also be transferred into the pixi-pack
tar or should they be installed manually after unpacking?is it possible to also bundle pip dependencies to this artifact
not at the moment, see https://github.com/Quantco/pixi-pack/issues/38 although i would suggest to build a conda package out of them anyway and put it on conda-forge as this tends to be the more stable approach from my experience
nodejs can be installed via conda but the npm dependencies have to be installed manually, will these also be transferred into the pixi-pack tar or should they be installed manually after unpacking?
pixi-pack only installs the conda packages, so the npm dependencies will not be included and need to be installed separately in some way.
i would suggest to just package them into a separate conda package and use --inject
, see https://github.com/Quantco/pixi-pack/tree/main/examples/webserver
Yep. "Just" turning conda packages into widely-respected (un)installer formats recognized by users (and their IT policies) is already a big lift, but "make anything relocatable," is... almost intractable, and indeed the purpose of the conda approach in the first place.
Even before pixi build
is a thing, a slightly-more-than-minimum viable constructor
example should likely show how to inject a simple, pixi run rattler-build
output into an installer build with little more than cat
.
Problem description
Hello Pixi devs,
Conda constructor
I use Conda constructor for creating installations that can then be used on air gapped machines (no internet).
A very simple overview of how constructor works is as follows
environment.yml
or directly in theconstruct.yml
)These installations can then be used to create an offiline installation on any machine. The installation is a conda installer (customized since it has all the dependencies also packaged as part of the installation) that can then be installed using the same cli used to install miniconda.
Enhancement proposal for
pixi
Suggestion/Enhancement here would be that pixi also support such a similar mechanism. From what i have looked all/most of the necessary plumbing is already there.
A rough idea of how this might work is as follows
pixi construct
creates a.pixi
folder with all the dependencies per environment storedThere is also another tool from conda called conda pack but this doesn't offer the same flexiblity/robustness as conda constructor since conda pack more or less creates an archive out of the existing env to make it portable and doesn't offer any way of customizing. In addition conda pack is not under active development.
Thanks!