Support private virtualenvs and locks for workspace members

idlsoft commented 5 months ago

Workspace members currently share a single uv.lock and .venv, which comes with a couple of side-effects:

there can't be two packages with conflicting dependencies, even if they are independent executables.
you can accidentally reference code from any workspace member without having to declare a dependency, and unit-tests or linters are unlikely to notice.

This is unlike most languages, but usually no big deal, and having one .venv is convenient for most use-cases.

But it isn't for everyone, and I think it would be useful to introduce a distinct kind of member, which:

will be excluded from the shared uv.lock/.venv
will have its own uv.lock/.venv, with only itself and its dependencies (workspace and otherwise).

I have tried to achieve this with one simple change, and it mostly works, but it's clunky and I think it could be much better.

Let's say we have this tree:

monorepo
├── apps
│   ├── app1
│   └── app2
└── libs
    ├── lib1
    └── lib2

And let's say we're ok with libraries using the default shared lock, but apps should not.

The workspace could be defined as

[tool.uv.workspace]
members = ["libs/*"]
private-members = ["apps/*"]

This is both explicit and efficient because workspace discovery won't have to read individual pyproject.toml files. Wrote a basic implementation with a sample workspace.

Hope this makes sense.

mjclarke94 commented 5 months ago

This would be a really clean solution to the refactor I'm trying to do.

It also seems like a lightweight change that unlocks a lot of the what Pants can do (finding the minimum set of dependencies required for each project, although that works per file and is a lot more powerful).

charliermarsh commented 4 months ago

How does this differ from excluding the projects from the workspace?

idlsoft commented 4 months ago

How does this differ from excluding the projects from the workspace?

Excluded projects cannot use workspace dependencies, these can. And they're allowed to have conflicting dependencies because they use private uv.lock and .venv directories and aren't included in members_as_requirements

charliermarsh commented 4 months ago

It's an interesting idea... I understand the motivation but needs some thought / discussion before committing to it.

idlsoft commented 4 months ago

needs some thought / discussion before committing to it.

Yesterday I couldn't resist trying to implement it, it turned out to be a lot more straightforward than I expected. Created https://github.com/astral-sh/uv/pull/4610 as a POC.

idlsoft commented 4 months ago

It's an interesting idea... I understand the motivation but needs some thought / discussion before committing to it.

Rye had a similar issue raised a little while ago.

pvardanis commented 3 months ago

@idlsoft @charliermarsh that would be really nice to have as a feature, especially for us that work on monorepos. I'm working on such a monorepo where some of the members of the workspace have conflicting dependencies (that is common for ML libraries), but also some common ones. That would require having separate venvs/locks for those packages but I'd still like to have common dependencies for all of the packages on the workspace level vs. not having a workspace at all, so I don't have repeated/common dependencies specified in multiple pyproject.toml files.

The example repo on your PR is exactly what I was looking for: https://github.com/astral-sh/uv/tree/5f2d0323d6330395dc8b62c6a126a234176165f0/scripts/workspaces/shared-libs.

astral-sh / uv

Support private virtualenvs and locks for workspace members #4574