Open 183amir opened 3 months ago
Yeah I saw this as well. Not sure what's causing this, we should have a closer look!
The cause is the exported CONDA_PREFIX
variable in the ~/.pixi/bin/uv
bash script. uv
uses this CONDA_PREFIX
variable to determine the currently activated conda environment. I have removed that line and now uv works but would nice if this was fixed here.
I'm not sure about these things, but shouldn't python
be a runtime dependency for uv? As you need this for using uv, and I would suppose adding that would install it into the global prefix? https://github.com/conda-forge/uv-feedstock
Pinging @wolfv @ruben-arts who should know more about these things :)
Edit: and @bollwyvl because he is a maintainer as well
Yeah, @183amir you correctly identified the problem. I am unsure how we should fix this.
We have some other problems with global install so I think over time we want to introduce more configuration to the "global" installs - and one of the config options could be about the CONDA_PREFIX.
Hypothetically it might also be nice to have some shared configuration for all pixi users that covers some cases like this (either as annotations in the packages or as some external file).
The initial stab at uv
had python
as a dependency, but on review in staged-recipes
, it was noted that it was shipping one file to $PREFIX/bin
, and didn't link against python. From a conda-forge perspective, this meant we could get away with ~5 {arch} * {platform}
builds instead of the * {py}
matrix factor for ~25 builds.
We can adopt something like the Arch pattern, where they have a noarch python-uv
which depends on uv
(the binary), potentially even in the same build (e.g. only build on linux x 64 x py3.12
), but could run with any supported conda-forge
python.
Is there any reason why symbolic links are not used to export global installed bins?
And I tried using a symbolic link to uv
and it seems to resolve this issue.
I'm not sure about these things, but shouldn't python be a runtime dependency for uv? As you need this for using uv, and I would suppose adding that would install it into the global prefix? https://github.com/conda-forge/uv-feedstock
No, uv doesn't need python to run. python is not a runtime dependency of uv. uv can modify any python installation by installing pypi packages in there. It doesn't matter if the python installation is the system's python, a conda environment, or a virtual environment.
Because you may have several python installations like the system one and a conda env, uv has some heuristics to detect where your currently activated python env is. Then, when you run commands like uv pip install flask
, it will install packages in that environment.
For conda environments, uv detects them through the CONDA_PREFIX
env variable. So when I run something like:
conda activate my_env
uv pip install flask
uv will look at the CONDA_PREFIX
env variable and find the python installation of the my_env
conda environment and then installs flask there.
Now the problem with installing uv with pixi is that pixi always suppresses this CONDA_PREFIX
env variable to point the uv environment. This happens inside ~/.pixi/bin/uv
:
#!/bin/sh
export PATH="~/.pixi/envs/uv/bin:${PATH}"
export CONDA_PREFIX="~/.pixi/envs/uv"
"~/.pixi/envs/uv/bin/uv" "$@"
so when I do:
conda activate my_env # conda activate exports a CONDA_PREFIX env variable
~/.pixi/bin/uv pip install flask
the CONDA_PREFIX
variable doesn't point to the my_env conda environment anymore but points to export CONDA_PREFIX="~/.pixi/envs/uv"
(from the bash file: ~/.pixi/bin/uv
) and this breaks uv's python discovery: https://github.com/astral-sh/uv?tab=readme-ov-file#python-discovery
My sugesstion is not to add this line:
export CONDA_PREFIX="~/.pixi/envs/uv"
in the generated bash file when you see the user is installing uv
with pixi which is what @wolfv suggested as well:
Hypothetically it might also be nice to have some shared configuration for all pixi users that covers some cases like this (either as annotations in the packages or as some external file).
Yeah @183amir, I get what your are saying, and I agree that holds true for most use-cases. I'm just concerned about the use-case you want to use uv
as a way to globally manage venvs
, without having a global python interpreter installed. Without having python in the uv
conda environment this won't work for the uv pip x
commands, unless you a running in an activated environment in some way.
On the flip-side by keeping the CONDA_PREFIX
this will shortcircuit detection in un-activated pixi folders without manually setting the VIRTUAL_ENV
environment variable.
I'm just concerned about the use-case you want to use uv as a way to globally manage venvs, without having a global python interpreter installed.
If there is no global python interpreter installed, isn't it possible to install with pixi? like pixi global install python
? Or something like pixi global inject uv python
(if it gets implemented).
On the flip-side by keeping the CONDA_PREFIX this will shortcircuit detection in un-activated pixi folders without manually setting the VIRTUAL_ENV environment variable.
I am not sure what you mean here. If the CONDA_PREFIX
is set to the uv
global conda env, uv
will always try to install packages there. What is the flip-side here?
If you guys are ok with adding an exception for uv
to skip inserting the CONDA_PREFIX
line in the generated bash file, I can work on a pull request.
I am not sure what you mean here. If the
CONDA_PREFIX
is set to theuv
global conda env,uv
will always try to install packages there. What is the flip-side here?
There might not be globally activated env. The flip-side I mean is more in regards to if that variable is set uv will always use that python, so if there is a python included next to the uv env, it will prefer that even when running from a pixi environment for example. So if you global inject python next to uv, which would require the CONDA_PREFIX
again, it will prefer that python. I'm just going from the documentation here.
If there is no global python interpreter installed, isn't it possible to install with pixi? like pixi global install python? Or something like pixi global inject uv python (if it gets implemented).
I don't think uv would find it. I believe this is the discovery function: https://github.com/astral-sh/uv/blob/e0ad649c7449b7f54c0053e409ac6e7bf18a6f68/crates/uv-toolchain/src/discovery.rs#L204C4-L204C40
If you guys are ok with adding an exception for
uv
to skip inserting theCONDA_PREFIX
line in the generated bash file, I can work on a pull request.
I'm fine with this, maybe initially as an option to global install? Not sure what @wolfv thinks.
This could also be in a activation script in hte uv
feedstock - that would say something like if $CONDA_PREFIX include ".pixi/bin" then unset CONDA_PREFIX fi
.
Then we don't need to add any special rules into pixi. WDYT @183amir?
Thanks btw!
This could also be in a activation script in hte uv feedstock - that would say something like if $CONDA_PREFIX include ".pixi/bin" then unset CONDA_PREFIX fi.
That would still not work because uv
needs access to the CONDA_PREFIX
variable to find the currently activated env. unsetting it would not recover the previous CONDA_PREFIX
variable.
BTW, I have the same issue with zsh
now. I have installed zsh
with pixi global install zsh
and now I have a CONDA_PREFIX
env variable set by default in my shell which breaks programs who rely on the CONDA_PREFIX
env variable.
I discussed this topic with @baszalmstra, and we identified that the main use case for setting CONDA_PREFIX
are activation scripts of the package that might depend on it. We could stop setting CONDA_PREFIX
if the package doesn't have an activation script. That would help with uv
and zsh
, neither of those packages has an activation script.
What do you think @wolfv?
The reason I proposed a symbolic link solution is that simply skipping activation scripts didn't fully address my issue. Another challenge I encountered was installing gridtk using pixi global install
, which causes the tool to break. Specifically, the installed gridtk
script (~/.pixi/bin/gridtk
) ends up with a modified PATH
environment variable, and this disrupts binary resolution when submitting jobs.
For instance, when I run:
gridtk submit ... python train.py
python
is not selected from my current PATH
; instead, it is sourced from the gridtk
environment.
This is really useful @183amir, thanks!
The PATH
variable is also set in the script. Like the CONDA_PREFIX
we could just not set it, if there's no activation script. Do you then still encounter problems that are fixed by symlinks, but not with this approach?
@Hofer-Julian FYI that would not work in pixi shell, we rely on the PATH variable to find executables within the environment. But perhaps we can do something special for pixi global.
But perhaps we can do something special for pixi global.
Yes, that's what I had in mind
The only other potential issue I can think of is that some installed binaries might become broken without a symlink, particularly those that rely on the current shell. This is because installed binaries typically run under /bin/sh
, which is usually a symbolic link to /bin/bash
. However, if you’re using zsh
and the installed binary includes some kind of shell detection, it might not work correctly. That said, I haven't personally encountered this issue.
Checks
[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of pixi, using
pixi --version
.Reproducible example
fails with:
Issue description
I have installed
uv
withpixi
using the command:which installs uv in a conda environment in
~/.pixi/envs/uv
where the uv binary is installed in~/.pixi/envs/uv/bin/uv
and a bash script named~/.pixi/bin/uv
is created with the following content:but the exported
CONDA_PREFIX
variable breaksuv
since it is meant to work on currently activated conda environment.Expected behavior
I expect to be able to install and use uv with pixi global install. However due to the nature of uv and how pixi creates global installations, uv breaks. I am not sure this is a bug in uv or pixi but would be nice to be able to install uv with pixi.
Note that I am not trying to use uv to install packages in a pixi environment. I know that pixi integrates uv internally. I am trying to use uv to pip install some packages in a conda environment.