search

conda / rattler

Rust crates to work with the Conda ecosystem.
BSD 3-Clause "New" or "Revised" License
262 stars 55 forks source link

Implement layered package caches #43

Open baszalmstra opened 1 year ago

baszalmstra commented 1 year ago

Conda and Mamba both support having multiple cache directories. Rattler currently only supports a single directory. The "layered" cache can have multiple readable directories and only one writable directory. Conda/Mamba tries to write a magical file to all these directories at startup to determine which one is writable.

Rattler should also facilitate "layered" package caches. We could introduce a trait for PackageCache that is implemented for both the current implementation as well as a layered version.

gzm55 commented 5 months ago

any updates?

baszalmstra commented 5 months ago

This is not being worked on atm. Contributions are welcome!

kelvinou01 commented 1 month ago

@baszalmstra hey, I'm trying to work out a PR for this. Do we need to enforce the user to have strictly one writable cache directory?

  1. Looking at conda's config class docs: 
pkgs_dirs=dals(
    """
    The list of directories where locally-available packages are linked from at
    install time. Packages not locally available are downloaded and extracted
    into the first writable directory.
    """
)

Seems like multiple writable package caches are allowed. Packages are both downloaded and extracted to the first writable cache

  1. I think it makes sense for Rattler to allow multiple writable caches.

Consider the scenario where you have a multi-user linux machine with a system-wide rattler package cache. This system-wide cache is read-only to users, and only writable by the system admin. e.g. normal user alice has RW access to /home/alice/.cache/rattler/cache, but only R access to /.cache/rattler/cache (I think where this system-wide cache dir is located is arbitrary?).

If a system admin bob has their own local cache dir (e.g. /home/bob/.cache/rattler/cache), bob would now have two writable caches. Enforcing strictly one writable cache would break this.

wolfv commented 1 month ago

I think a user could have more than one writable cache, but we should probably write to the "first" one. If we would have a nice API to list all available package caches, and those that are writable, it would be nice to be able to show that to users (e.g. with pixi info).

@baszalmstra is also currently working on much improving the package cache concurrency.