Option/setting to manage virtualenvs centrally

DanCardin commented 9 months ago

I ideally dont .venv/ folders cluttering my project folders (not the least because they may not be safe to move, at least with venv)

In my personal (also rust) workflow tool (that i'd love to not have to maintain if uv could arrive at some of the same decisions 😆) there is a setting which when set to "central" puts all venvs in $XDG_DATA_HOME/<toolname>/..., and the path to the venv is determined by mirroring the path to the project. That is, ~/foo/bar/baz/pyproject.toml -> $XDG_DATA_HOME/uv/foo/bar/baz/.venv/.

By comparison to more traditional tools, poetry also (by default) will automatically create venvs in a central location. Although it puts all venvs in the same folder using a somewhat inscrutable hash to disambiguate projects of the same name.

This ^ feature sort of implies a few other mostly separate features in order to be useful:

uv activate (because it becomes impractical to self-activate, except by copy-pasting the path that gets printed)
which then implies some uv self init --shell zsh (or whatever), that gives you the shell integration to automatically activate through the CLI itself.
uv venv --delete or -d

zanieb commented 9 months ago

Thanks for the issue!

This is the kind of thing we plan to tackle in the future, i.e. when we build an opinionated workflow for environment management. It's not in scope right this second, but we'll revisit it :)

woutervh commented 9 months ago

By comparison to more traditional tools, poetry also (by default) will automatically create venvs in a central location. Although it puts all venvs in the same folder using a somewhat inscrutable hash to disambiguate projects of the same name.

IMHO: It's one of the really bad decisions of poetry. As a python contractor usually supporting other python developers, I really dislike any situation where devs have no idea where their virtualenv of their project is. Your executables are the most important thing in your project, hiding them in arcane directories should never be a default.

Activating a venv is an antipattern because it introduces state in your shell. I really hope it can be de-emphasized in the future.

DanCardin commented 9 months ago

Imo there are various positive sideeffects of the venv not being local, although I dont personally like their chosen algorithm for selecting it.

But ultimately, as long as the tool can know where the venv should be given the settings/invocation options, then there isn't generally a need to activate the venv, at least for uv itself to function. But it's the reality that many tools require VIRTUAL_ENV to be set to function properly, which either means activating the venv or routing all commands through a uv run (which might be nice interactively but isn't ideal for committed files imo).

ResRipper commented 9 months ago

I use Windows/Linux/Mac every day and synchronise my projects using OneDrive, having .venv in my project folder is such a nightmare, and that's why I started using pipenv. IMO venv should always be centralised as environments and packages are platform dependent, they are not part of the "content" of the project.

I'm using this plugin to manage and switch between different environments and don't have to know their location most of the time, but I do hope there is a standard for that.

hauntsaninja commented 9 months ago

Note uv currently appears to work if you make .venv file a symlink to your actual venv

If you don't have symlinks on your platform, this patch of uv may work for you by adding support for .venv files that point to the actual venv location: https://github.com/astral-sh/uv/issues/1578#issuecomment-1949911871

When uv does go in the higher level workflow direction, I'd advocate leaving a .venv symlink or file pointing to the central location. This makes it easy for IDEs and other tools to figure out where the environment is. This .venv file / symlink idea was discussed around the time PEP 704 was a thing and had fairly positive support

woutervh commented 9 months ago

@hauntsaninja Thanks for the tip.

I was annoyed that uv does not support the most common normal use-case of virtualenv ootb:

> virtualenv foo
> cd foo
> bin/pip install <package1>

To make it work, the symlink indeed works

> uv venv foo
> cd foo
> ln -s .  .env
> uv pip install <package2>

It would be nice if "uv venv" would make the link by default.

And this would be a nice solution for managing venvs centrally outside the project-folder.

@ResRipper and for syncing via onedrive, you can point .venv to one of the os-specific .venv-files

.venv -> .venv-linux
.venv-linux  --> /some/linux/path
.venv-mac  --> /some/mac/path
.venv-windows  --> /some/windows/path

matterhorn103 commented 7 months ago

Just to add my voice that this would be really really nice to have. Different tools seem to choose either one approach or the other, and it would be great if uv would do both as I like and use both in different situations:

For development situations where git is being used, having .venv there in the project's folder is convenient.

For other projects e.g. scientific ones that are in maybe shared or cloud or working folders, it's undesirable behaviour, and then working with conda is much smoother than with Python venvs. That's pretty much the only thing that keeps me using conda for some things (other than the different ecosystem, naturally).

My impression was that symlinks have poor portability, so like @DanCardin I'd prefer something like a simple -c switch to create the venv in an automatically determined central location, and simple automatic activation, e.g.:

~/example/foo >>> uv venv -c
Using Python 3.11.9 interpreter at: /usr/bin/python3
Creating virtualenv at: /home/jdoe/.local/share/uv/example/foo/.venv
Activate with: uv venv activate
~/example/foo >>> uv venv activate
Activating virtualenv at /home/jdoe/.local/share/uv/example/foo/.venv
(foo) ~/example/foo >>> deactivate
~/example/foo >>>

but with the addition that it would be cool to be able to also activate the venv by name from another location, with the search resolved intelligently and some ability to disambiguate; I could imagine that looking like:

~/some/folder >>> uv activate foo
Searching for virtualenvs at /home/jdoe/.local/share/uv/**/foo/.venv
Activating virtualenv at: /home/jdoe/.local/share/uv/example/foo/.venv
(foo) ~/some/folder >>> uv activate bar
Searching for virtualenvs at /home/jdoe/.local/share/uv/**/bar/.venv
error: virtualenv name is ambiguous! The following matches were found:
  1) /home/jdoe/.local/share/uv/Documents/bar/.venv
  2) /home/jdoe/.local/share/uv/project1/bar/.venv
disambiguate venvs with the same name using parents e.g. to activate 1) use:
  uv venv activate Documents/bar
(foo) ~/some/folder >>> uv activate project1/bar
Searching for virtualenvs at /home/jdoe/.local/share/uv/**/project1/bar/.venv
Activating virtualenv at: /home/jdoe/.local/share/uv/project1/bar/.venv
(bar) ~/some/folder >>>

matterhorn103 commented 5 months ago

So in addition to the "not wanting the venv to be stored in a folder that is backed up to the cloud" use case, I found another use case today:

Creating compiled binaries of Qt apps written with PySide using pyside6-deploy (a nuitka wrapper) isn't possible if a venv folder is present in the application folder

(In my view this is a short-sighted approach on the tool's part, but it's just another example of why it might be necessary to keep a venv elsewhere.)

callegar commented 2 months ago

When uv does go in the higher level workflow direction, I'd advocate leaving a .venv symlink or file pointing to the central location.

and

It would be nice if "uv venv" would make the link by default. And this would be a nice solution for managing venvs centrally outside the project-folder.

Even this makes things more complex than needed if you sync to the cloud across multiple systems, because the link might need to be to different places on different machines.

If only for compatibility and ease of migration from one tool to another, I would offer the possibility to do what pdm does. That tool offers either the option to have a local .venv or the option to have venvs stored all together in a centralized place.

Note that having the venvs all stored in a single place also makes it easier to apply deduplication tools on suitable filesystems.

PhilipVinc commented 2 months ago

@zanieb is this a feature you're considering for the short term, or it's not high priority?

zanieb commented 2 months ago

This is not something we're tackling immediately — there are a few workarounds and we are prioritizing work that affects more users.

hauntsaninja commented 2 months ago

Since it hasn't been mentioned on this issue yet:

For the high level project workflows, you can use UV_PROJECT_ENVIRONMENT to control venv location. Symlinking .venv also works
For low level uv pip/venv workflows, you can set UV_PYTHON / --python, or similarly symlink .venv as above

phi6ias commented 1 month ago

I would like to ask to extend the thought towards a centralised location for venvs and caches in the internal network of a company.

Besides reducing the amount of storage wasted on local hard drives, the aim would be to allow a developer to write a programme or script, add an extra entry to the script section in the (main) file, as described here, that points to the venv he created on the network share and have it circulated to colleagues as if it was a standalone (i.e. no installation required other than uv itself).

callegar commented 1 month ago

The .venv symlink is possibly not a good workaround for those who store their work in the cloud and use it on different platforms. Depending on the cloud-sync tool that you use, you may either:

get your symlink also synchronized "as is", forcing you to recreate the link in case the actual venv is in different locations on the different platforms (as it is often the case)
have your sync client broken enough to follow the link and sync the destination.

Often you can "exclude files by pattern" on sync clients, to relieve these problems, but it is not always handy. In these cases, the env variable might be a better stopgap.

fabienval commented 1 month ago

I too would be keen to have venv centrally stored due to onedrive not playing nice when trying to sync .venv.... I see people suggesting symlinks, but when we are on windows, is there a way to have similar behavior? I tried creating a shortcut named .venv but uv recreate a proper folder called .venv and install in there when I uv sync... ( We do not have permission to use tools like mklink)

hauntsaninja commented 1 month ago

Does the UV_PROJECT_ENVIRONMENT trick work for your use case? If not, you could try rebasing this patch https://github.com/astral-sh/uv/issues/1578#issuecomment-1949911871 and see if uv is willing to merge it

krstp commented 4 weeks ago

See my solution with .bashrc/.zshrc function uvactivate <named_venv> from a central local where I keep all my venvs: https://github.com/astral-sh/uv/issues/7898

callegar commented 2 weeks ago

Possibly, having venvs in a central location would also allow making --link-mode=symlink more robust. Currently, if you use it and then by mistake you clear the cache, all your venvs made with this link mode will be broken. If all the venvs are centralized, the cache clearing command could get an operating mode (maybe to be made the default) where the (centralized) environments are scanned and the stuff that has symlinks to is preserved in the cache.

YodaEmbedding commented 2 days ago

@callegar Poetry puts its virtualenvs in $XDG_CACHE_HOME/pypoetry/virtualenvs. Theoretically, poetry virtualenvs should be reproducible, so this is logical. However, cache clearing should be a safe operation that should not have adverse effects on startup performance.

Thus, a better location might be $XDG_STATE_HOME/pypoetry/virtualenvs, i.e. ~/.local/state/pypoetry/virtualenvs.

astral-sh / uv

Option/setting to manage virtualenvs centrally #1495