Annotate sources of requirements

palfrey commented 2 weeks ago

Summary

Fixes https://github.com/astral-sh/uv/issues/1343. This is kinda a first draft at the moment, but does at least mostly work locally (barring some bits of the test suite that seem to not work for me in general).

Test Plan

Mostly running the existing tests and checking the revised output is sane

Outstanding issues

Most of these come down to "AFAIK, the existing tools don't support these patterns, but uv does" and so I'm not sure there's an existing good answer here! Most of the answers so far are "whatever was easiest to build"

[x] ~~Is "-r pyproject.toml" correct? Should it show something else or get skipped entirely~~ No it wasn't. Fixed in 3044fa8b86791369a3c468ced585753428e70b1b
[ ] If the requirements file is stdin, that just gets skipped. Should it be recorded?
[ ] Overrides get shown as "--override". Correct?
[x] Some of the tests (e.g. dependency_excludes_non_contiguous_range_of_compatible_versions) make assumptions about the order of package versions being outputted, which this PR breaks. I'm not sure if the text is fairly arbitrary and can be replaced or whether the behaviour needs fixing? - fixed by removing the custom pubgrub PartialEq/Hash
[ ] Are all the TrackedFromStr et al changes needed, or is there an easier way? I don't think so, I think it's necessary to track these sort of things fairly comprehensively to make this feature work, and this sort of invasive change feels necessary, but happy to be proved wrong there :)
[x] If you have a requirement coming in from two or more different requirements files only one turns up. I've got a closed-source example for this (can go into more detail if needed), mostly consisting of a complicated set of common deps creating a larger set. It's a rarer case, but worth considering. 042432b200395d498b4517121a9fb25f1bbda8ca
[ ] Doesn't add annotations for setup.py yet
- This is pretty hard, as the correct location to insert the path is crates/pypi-types/src/metadata.rs's parse_pkg_info, which as it's based off a source distribution has entirely thrown away such matters as "where did this package requirement get built from". Could add "built package name" as a dep, but that's a little odd.

ThiefMaster commented 2 weeks ago

I just tested it with the requirements files of https://github.com/indico/indico/, and for requirements.{in,txt} it works great - no changes beyond the comments on top! :)

For requirements.dev.{in,txt}, however, I noticed one bug: Instead of -c requirements.txt it uses -r requirements.txt in the source comments. That aside, it seems to do everything fine there as well.

zanieb commented 2 weeks ago

Exciting! I'm not a good review for this one, but I wonder if it could be used to address https://github.com/astral-sh/uv/issues/1854 next?

palfrey commented 2 weeks ago

I just tested it with the requirements files of https://github.com/indico/indico/, and for requirements.{in,txt} it works great - no changes beyond the comments on top! :)

For requirements.dev.{in,txt}, however, I noticed one bug: Instead of -c requirements.txt it uses -r requirements.txt in the source comments. That aside, it seems to do everything fine there as well.

9655e9139d7ea2ea02d979fcc255d45542aa287f should fix that.

palfrey commented 2 weeks ago

@charliermarsh I'm seeing some test failures on Windows, and they appear to be related to path redactions in insta. There's some nice work for doing that cross-platform in crates/uv/tests/common/mod.rs, but I'm needing it for things like crates/requirements-txt/src/lib.rs and that would be a circular dependency. Any thoughts on how to approach sorting those out? I don't have an easy way to do Windows work sadly, so this might be a bit fiddly!

ThiefMaster commented 2 weeks ago

https://github.com/astral-sh/uv/commit/9655e9139d7ea2ea02d979fcc255d45542aa287f should fix that.

Indeed, the output is exactly what I expect now!

```diff [adrian@claptrap:~/dev/indico/src:master *$]> git diff diff --git a/docs/requirements.txt b/docs/requirements.txt index af095861bf..e16ac62be7 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -1,9 +1,5 @@ -# -# This file is autogenerated by pip-compile with Python 3.12 -# by the following command: -# -# pip-compile --strip-extras docs/requirements.in -# +# This file was autogenerated by uv via the following command: +# uv pip compile -o docs/requirements.txt docs/requirements.in alabaster==0.7.16 # via sphinx babel==2.14.0 diff --git a/requirements.dev.txt b/requirements.dev.txt index 6146f0535c..3fdd46a5bd 100644 --- a/requirements.dev.txt +++ b/requirements.dev.txt @@ -1,9 +1,5 @@ -# -# This file is autogenerated by pip-compile with Python 3.12 -# by the following command: -# -# pip-compile --strip-extras requirements.dev.in -# +# This file was autogenerated by uv via the following command: +# uv pip compile -o requirements.dev.txt requirements.dev.in aiosmtpd==1.4.5 # via pytest-localserver alabaster==0.7.16 @@ -125,6 +121,8 @@ parso==0.8.4 # pyquotes pep8-naming==0.13.3 # via -r requirements.dev.in +pip==24.0 + # via pip-tools pip-tools==7.4.1 # via -r requirements.dev.in plantweb==1.2.1 @@ -190,6 +188,10 @@ ruff==0.4.1 # via -r requirements.dev.in schemainspect==3.1.1663587362 # via migra +setuptools==69.5.1 + # via + # flake8-quotes + # pip-tools six==1.16.0 # via # -c requirements.txt @@ -267,7 +269,3 @@ werkzeug==3.0.2 # pytest-localserver wheel==0.43.0 # via pip-tools - -# The following packages are considered to be unsafe in a requirements file: -# pip -# setuptools diff --git a/requirements.txt b/requirements.txt index 32521e6a04..a8ec374820 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,9 +1,5 @@ -# -# This file is autogenerated by pip-compile with Python 3.12 -# by the following command: -# -# pip-compile --strip-extras -# +# This file was autogenerated by uv via the following command: +# uv pip compile -o requirements.txt requirements.in alembic==1.13.1 # via # -r requirements.in ```

jardarbo commented 1 week ago

@palfrey Very nice of you to look at this!

Testing your version with the following pyproject.toml:

$ cat pyproject.toml
[project]
name = "uv_test"
dynamic = ["version"]
dependencies = ["click"]

Running uv pip compile with your version yields the following:

$ uv pip compile pyproject.toml
Resolved 1 package in 4ms
# This file was autogenerated by uv via the following command:
#    uv pip compile pyproject.toml
click==8.1.7
    # via -r pyproject.toml

Running pip-compile (https://github.com/jazzband/pip-tools) yields:

$ pip-compile pyproject.toml
#
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
#    pip-compile pyproject.toml
#
click==8.1.7
    # via uv_test (pyproject.toml)

The difference is # via -r pyproject.toml vs. # via uv_test (pyproject.toml).

I don't know if you think we should match the output of pip-compile, but it makes it easier to mix the usage of uv pip compile and pip-compile.

Running this on Ubuntu on Windows 11 WSL.

ThiefMaster commented 1 week ago

+1 on matching the pip-compile output, it makes transitioning from pip-tools to uv much more straightforward (as you immediately see that nothing changed), and -r pyproject.toml seems weird anyway (unless pip install -r pyproject.toml is valid which I'd find surprising)

palfrey commented 1 week ago

The difference is # via -r pyproject.toml vs. # via uv_test (pyproject.toml).

I don't know if you think we should match the output of pip-compile, but it makes it easier to mix the usage of uv pip compile and pip-compile.

I didn't investigate this path much, but had seen -r pyproject.toml and wasn't sure offhand how this was meant to be handled. Now I know, 3044fa8b86791369a3c468ced585753428e70b1b fixes this.

jardarbo commented 1 week ago

Now I know, https://github.com/astral-sh/uv/commit/3044fa8b86791369a3c468ced585753428e70b1b fixes this.

Wow, this was quick @palfrey, thanks! This does indeed make output same as pip-compile.

```bash $ uv pip compile pyproject.toml Resolved 1 package in 4ms # This file was autogenerated by uv via the following command: # uv pip compile pyproject.toml click==8.1.7 # via uv-test (pyproject.toml) ```

palfrey commented 1 week ago

@charliermarsh I've just updated to latest main. I'd regard this as done, except for the Windows test failures, which I'm not sure how to fix (see previous comment on this). However, that was painful to update, so a review of this very soon would be appreciated!

If anyone else has bright ideas about how to fix the Windows issue, I'm all ears.

charliermarsh commented 1 week ago

Thank you @palfrey and sorry for the painful rebase. I'd been holding off on reviewing this until the sources PR merged, since I knew it would lead to changes here, but I should've said so earlier. I will own reviewing now.

palfrey commented 1 week ago

If anyone else has bright ideas about how to fix the Windows issue, I'm all ears.

safe_filter_path fixes the regex issues (mostly by copy/paste from crates/uv/tests/common/mod.rs) but there's still some issues, both with a lack of replacement in some cases, and a difference in output on Windows v.s. Linux that I can't easily resolve without a local Windows dev box.

palfrey commented 4 days ago

I will own reviewing now.

How's that going? I'm seeing a new fun set of conflicts, but didn't want to resolve those until the next bit of reviewing was done ideally.