Add basic `tool.uv.sources` support

[konstin]

Introduction

PEP 621 is limited. Specifically, it lacks

• Relative path support
• Editable support
• Workspace support
• Index pinning or any sort of index specification

The semantics of urls are a custom extension, PEP 440 does not specify how to use git references or subdirectories, instead pip has a custom stringly format. We need to somehow support these while still stying compatible with PEP 621.

tool.uv.source

Drawing inspiration from cargo, poetry and rye, we add tool.uv.sources or (for now stub only) tool.uv.workspace:

[project]
name = "albatross"
version = "0.1.0"
dependencies = [
  "tqdm >=4.66.2,<5",
  "torch ==2.2.2",
  "transformers[torch] >=4.39.3,<5",
  "importlib_metadata >=7.1.0,<8; python_version < '3.10'",
  "mollymawk ==0.1.0"
]

[tool.uv.sources]
tqdm = { git = "https://github.com/tqdm/tqdm", rev = "cc372d09dcd5a5eabdc6ed4cf365bdb0be004d44" }
importlib_metadata = { url = "https://github.com/python/importlib_metadata/archive/refs/tags/v7.1.0.zip" }
torch = { index = "torch-cu118" }
mollymawk = { workspace = true }

[tool.uv.workspace]
include = [
  "packages/mollymawk"
]

[tool.uv.indexes]
torch-cu118 = "https://download.pytorch.org/whl/cu118"

See docs/specifying_dependencies.md for a detailed explanation of the format. The basic gist is that project.dependencies is what ends up on pypi, while tool.uv.sources are your non-published additions. We do support the full range or PEP 508, we just hide it in the docs and prefer the exploded table for easier readability and less confusing with actual url parts.

This format should eventually be able to subsume requirements.txt's current use cases. While we will continue to support the legacy uv pip interface, this is a piece of the uv's own top level interface. Together with uv run and a lockfile format, you should only need to write pyproject.toml and do uv run, which generates/uses/updates your lockfile behind the scenes, no more pip-style requirements involved. It also lays the groundwork for implementing index pinning.

Changes

This PR implements:

• Reading and lowering project.dependencies, project.optional-dependencies and tool.uv.sources into a new requirements format, including:
  • Git dependencies
  • Url dependencies
  • Path dependencies, including relative and editable
• pip install integration
• Error reporting for invalid tool.uv.sources
• Json schema integration (works in pycharm, see below)
• Draft user-level docs (see docs/specifying_dependencies.md)

It does not implement:

• No pip compile testing, deprioritizing towards our own lockfile
• Index pinning (stub definitions only)
• Development dependencies
• Workspace support (stub definitions only)
• Overrides in pyproject.toml
• Patching/replacing dependencies

One technically breaking change is that we now require user provided pyproject.toml to be valid wrt to PEP 621. Included files still fall back to PEP 517. That means pip install -r requirements.txt requires it to be valid while pip install -r requirements.txt with -e . as content falls back to PEP 517 as before.

Implementation

The pep508 requirement is replaced by a new UvRequirement (name up for bikeshedding, not particularly attached to the uv prefix). The still existing pep508_rs::Requirement type is a url format copied from pip's requirements.txt and doesn't appropriately capture all features we want/need to support. The bulk of the diff is changing the requirement type throughout the codebase.

We still use VerbatimUrl in many places, where we would expect a parsed/decomposed url type, specifically:

• Reading core metadata except top level pyproject.toml files, we fail a step later instead if the url isn't supported.
• Allowed Urls.
• PackageId with a custom CanonicalUrl comparison, instead of canonicalizing urls eagerly.
• PubGrubPackage: We eventually convert the VerbatimUrl back to a Dist (Dist::from_url), instead of remembering the url.
• Source dist types: We use verbatim url even though we know and require that these are supported urls we can and have parsed.

I tried to make improve the situation be replacing VerbatimUrl, but these changes would require massive invasive changes (see e.g. https://github.com/astral-sh/uv/pull/3253). A main problem is the ref VersionOrUrl and applying overrides, which assume the same requirement/url type everywhere. In its current form, this PR increases this tech debt.

I've tried to split off PRs and commits, but the main refactoring is still a single monolith commit to make it compile and the tests pass.

Demo

Adding https://gist.githubusercontent.com/konstin/68f3e5ea5e8f1a30a0c9096ae72925c1/raw/d1ae3b85d5a0f3c4bb562c7dfd38c3dafc04ec08/pyproject.json as json schema (v7) to pycharm for pyproject.toml, you can try the IDE support already:

dove.webm

[konstin]

• #3277
• #3263 👈
• main

[charliermarsh]

Another round of comments. I don't really expect to ask for any major changes, but I still have two slightly larger topics to resolve:

1. I still kinda feel like requirements-txt should not rely on the uv-specific requirement types.
2. I'm trying to understand what a better setup would look like to allow us to avoid all the URL reconstruction and deconstruction throughout the codebase.

[konstin]

I still kinda feel like requirements-txt should not rely on the uv-specific requirement types.

Fixed by introducing UnresolvedRequirementSpecification and UnresolvedRequirement in distribution-types. It's a bit odd at first because requirements-txt depends on distribution-types, but it avoid cyclical crate dependencies and does the split.