search

astral-sh / uv

An extremely fast Python package and project manager, written in Rust.
https://docs.astral.sh/uv
Apache License 2.0
27.56k stars 793 forks source link

Usage in docker: `error: Project virtual environment directory /app/.venv cannot be used because it is not a compatible environment but cannot be recreated because it is not a virtual environment #9423

Open ben-xD opened 12 hours ago

ben-xD commented 12 hours ago

Thanks for the detailed docker docs! I'm trying to use uv in docker compose for local development. Trying to bind mount my application code and an anonymous volume for app/.venv. This is similar to the approach shown in docs with docker run which aims to avoid overwriting host vs docker container venv files, and keep them separate.

Sadly, uv sync inside the container errors with:

error: Project virtual environment directory /app/.venv cannot be used because it is not a compatible environment but cannot be recreated because it is not a virtual environment

This might be because uv tries to replace the .venv folder, but it can't since it's a volume?

Docker compose settings

services:
  reproduce:
    build:
      context: .
      dockerfile: Dockerfile
    entrypoint: "tail -f /dev/null"
    volumes:
      - ./:/app
      - /app/.venv

Details

Reproducible setup

Following documentation's docker run

I also tried following the docs, by running:

docker run -it --rm --volume .:/app --volume /app/.venv uv-reproduce-error-reproduce bash

Then running

uv sync

but i get the same error:

root@ea61fab74354:/app# uv sync
Using CPython 3.11.10
error: Project virtual environment directory `/app/.venv` cannot be used because it is not a compatible environment but cannot be recreated because it is not a virtual environment
ben-xD commented 12 hours ago

The docker compose watch feature isn't ideal though because it the python dependencies are redownloaded whenever the image needs to be rebuild - it doesn't cache .venv, just ignores it.

I also can't enter the container shell to run commands interactively (it really helps experimenting with container/linux tools used in CI quickly). Also, the sync is 1 way: i cant make changes inside the container.

zanieb commented 11 hours ago

Thank you for the reproduction!

The reason you get that error is because the .venv directory exists, but is not a virtual environment. This is a safeguard to avoid mutating target directories that we should not. We might want to relax that to allow mutating an empty directory, but this conflicts with our removal of the directory to create a virtual environment:

root@8ceb311d28c1:/app# uv venv
Using CPython 3.11.10
Creating virtual environment at: .venv
uv::venv::creation

  x Failed to create virtualenv
  `-> failed to remove directory `/app/.venv`: Resource busy (os error 16)

the python dependencies are redownloaded whenever the image needs to be rebuild - it doesn't cache .venv, just ignores it.

Perhaps you're looking to use a cache mount to store dependency downloads?

Also, the sync is 1 way: i cant make changes inside the container.

Here the idea is that you would be able to modify the pyproject.toml with uv add and have it reflected inside and outside of the container?

ben-xD commented 11 hours ago

Thanks for your reply! Yes it makes sense that it doesn't yet work but I can't find another way of isolating the 2 python environments (docker/linux vs. host/macOS) to avoid needing to download/reinstall everything when switching between testing my python app in my docker and host environments. I can't "exclude" the .env.

I don't think cache mount would work since I want to update the cache whilst inside the container like with docker exec -it $container_name bash? Cache mounts are for build time?

Here the idea is that you would be able to modify the pyproject.toml with uv add and have it reflected inside and outside of the container?

The 2 way sync is not super important tbh. But yea, it would be ideal if uv add updates the lock file, and that's synced to my host file. system. But definitely syncing from host to linux. It would be nice if uv sync just downloads the package that was missing.

zanieb commented 11 hours ago

Okay confusing, we do ignore empty directories but it's different in this case? Presumably because it's a mounted volume? If I run uv venv twice, we create it:

root@8ceb311d28c1:/app# uv venv -v
DEBUG uv 0.5.4
DEBUG Found project root: `/app`
DEBUG No workspace root found, using project root
DEBUG Reading Python requests from version file at `/app/.python-version`
DEBUG Using Python request `3.11` from version file at `.python-version`
DEBUG Searching for Python 3.11 in managed installations or search path
DEBUG Searching for managed installations at `/root/.local/share/uv/python`
DEBUG Found managed installation `cpython-3.11.10-linux-aarch64-gnu`
DEBUG Found `cpython-3.11.10-linux-aarch64-gnu` at `/root/.local/share/uv/python/cpython-3.11.10-linux-aarch64-gnu/bin/python3.11` (managed installations)
Using CPython 3.11.10
Creating virtual environment at: .venv
DEBUG Removing existing directory
uv::venv::creation

  x Failed to create virtualenv
  `-> failed to remove directory `/app/.venv`: Resource busy (os error 16)
root@8ceb311d28c1:/app# uv venv -v
DEBUG uv 0.5.4
DEBUG Found project root: `/app`
DEBUG No workspace root found, using project root
DEBUG Reading Python requests from version file at `/app/.python-version`
DEBUG Using Python request `3.11` from version file at `.python-version`
DEBUG Searching for Python 3.11 in managed installations or search path
DEBUG Searching for managed installations at `/root/.local/share/uv/python`
DEBUG Found managed installation `cpython-3.11.10-linux-aarch64-gnu`
DEBUG Found `cpython-3.11.10-linux-aarch64-gnu` at `/root/.local/share/uv/python/cpython-3.11.10-linux-aarch64-gnu/bin/python3.11` (managed installations)
Using CPython 3.11.10
Creating virtual environment at: .venv
DEBUG Ignoring empty directory
zanieb commented 11 hours ago

I don't think cache mount would work since I want to update the cache whilst inside the container like with docker exec -it $container_name bash? Cache mounts are for build time?

I see. I was responding specifically to "whenever the image needs to be rebuild" — you could also mount the cache at runtime — though I'm not sure it'd be safe to concurrent access since we can't lock files across the host and container.

But yea, it would be ideal if uv add updates the lock file, and that's synced to my host file. system.

This sort of works, but Docker syncs the the virtual environment back to the host and breaks things. It seems feasible with some work.

zanieb commented 10 hours ago

I'm not sure what the best next step here is. Perhaps you can use UV_PROJECT_ENVIRONMENT to try to move the virtual environment outside of the project directory while in the container to avoid collisions.

ben-xD commented 10 hours ago

That's great! Thanks a lot! I was trying to use VIRTUAL_ENV but that was being ignored.

zanieb commented 9 hours ago

Does that solve all your problems here?