astral-sh / uv

An extremely fast Python package and project manager, written in Rust.
https://docs.astral.sh/uv
Apache License 2.0
26.04k stars 762 forks source link

Document strategies for sharing the cache #5611

Open zanieb opened 3 months ago

zanieb commented 3 months ago

Per https://github.com/astral-sh/uv/issues/5581#issuecomment-2258832686, sharing the cache across users can be useful for system-wide uv installations.

stas00 commented 3 months ago

Here is the summary so far of what needs to be done to make uv work with a shared cache - this helps to save a ton of disk space and inodes when there are many users on a system and files can be shared.

  1. add to ~/.bashrc or ~/.profile of each user:
umask 000 # file creation so that all users's files are a+rw or a+rwx
export UV_LINK_MODE=copy # hardlinking doesn't work with different owners
  1. Then create a shared cache folder:
    sudo mkdir -p /data/cache/uv
    sudo chmod a+rwx /data/cache/uv
  2. Then each user redirecting their local cache to the shared one:
    rm -r ~/.cache/uv
    ln -s /data/cache/uv ~/.cache/uv

and this requires uv>=0.2.32 to work

Done.

zanieb commented 3 months ago

Thanks! If you're setting environment variables anyway, you can use UV_CACHE_DIR and skip the symbolic link.

stas00 commented 3 months ago

oh, that's super-useful, then I don't have to go after users and making sure they installed a symlink.

Thank you very much, @zanieb!

paveldikov commented 3 months ago

Worth mentioning (or even discussing!) the security implications of this. (cache poisoning can be a huge problem)

MattiasDC commented 3 months ago

If you can't or don't want to use umask (as it will change behavior it for all files you create, not only uv cache). You can also use ACL rules on the cache directory, which overrides umask configuration.

rabin-io commented 1 day ago

What if you have many users WRITING to the cache? e.g. starting from a clean folder, and invoking many CI/test jobs at the same time which might want to download their files.