search

astral-sh / uv

An extremely fast Python package and project manager, written in Rust.
https://docs.astral.sh/uv
Apache License 2.0
20.09k stars 598 forks source link

uv cache populate / export cache #6323

Open adriangb opened 3 weeks ago

adriangb commented 3 weeks ago

It would be nice to have the ability to pre-cache dependencies. In particular I would find this useful in a monorepo context to:

  1. Have a docker layer where I populate the cache with all dependencies.
  2. In another service specific layer I do a --cache-from or similar from this cache layer and tell uv to use the cache that was built in that layer.

This is important because if I have services with the following deps:

tensorflow
some-small-package

And

tensorflow
some-other-small-package

If I don't have a "common" step where I download tensorflow I'd end up downloading it twice (once for each service) which is wasteful. Yes you can parallelize it but IMO efficient serial >> inefficient parallel.

Alternatively something like https://github.com/python-poetry/poetry/issues/5983 and functionality in uv to populate that folder of wheels could do the trick.

charliermarsh commented 3 weeks ago

What's the typical mechanism that you'd use to ensure that those two layers use the same TensorFlow version?

adriangb commented 3 weeks ago

Currently it's a PITA with all package managers that I know of that support monorepos.

Hence why I think something like:

FROM python:3.12-slim-bullseye AS deps
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
COPY uv.lock .
RUN --mount=type=cache,target=/root/.cache/uv \
  uv cache populate

FROM python:3.12-slim-bullseye AS service
COPY --from=ghcr.io/astral-sh/uv:latest /uv /bin/uv
COPY . /app
RUN --mount=from=deps,source=/root/.cache/uv,target=/root/.cache/uv \
  uv install --package service

Would be nice.

samypr100 commented 3 weeks ago

I'd say this is similar in the venue/spirit of https://github.com/astral-sh/uv/issues/1681 / https://github.com/astral-sh/uv/issues/3163 as populating the cache will involve downloading and possibly a build step which is a common workflow achieved with pip wheel or download.