search

conda-incubator / conda-store

Data science environments, for collaboration. ✨
https://conda.store
BSD 3-Clause "New" or "Revised" License
142 stars 46 forks source link

[ENH] - Support hash-only build paths #678

Closed nkaretnikov closed 4 months ago

nkaretnikov commented 9 months ago

Feature description

Introduce "Nix-like" hashes that are fixed-size. We currently have two build-key (build-key is the environment build part of the build-path) types: the legacy one and truncated (with a timestamp). In this new scheme, it'll be a completely new build-path, which is fixed-size and not depend on user-provided data, like in Nix. See here https://github.com/conda-incubator/conda-store/pull/652#discussion_r1390512691

Value and/or benefit

Not running into build_path limits, better control over build_path length (always fixed-size, only depends on conda-store root).

Anything else?

This is a big change. It requires thinking about collisions. Ideally, we want to share the same envs between multiple people, so this requires thinking about resource sharing. I have details on this in the link above. We might want to hash the lockfile as opposed to CondaSpecification as currently.

The proposed format is:

store_directory / hash / <env contents>
nkaretnikov commented 9 months ago

Related: v2 (truncated) build_key has landed as part of https://github.com/conda-incubator/conda-store/pull/652