get sdist requires from egg-info

[Avasam]

CC @AlexWaygood This was the idea I mentioned.

When the requirements can't be found in requires_dist, fallback to searching in *.egg-info. All without executing any arbitrary code.

I move the f"Expected dependency {req} to be present in the allowlist" error higher so it can fail earlier and leave the network requests last.

It's my first time playing around with downloading and manipulating archives in python. Let me know what I can do better here.

I tested the idea with pycocotools (best case scenario) click-default-group (on my windows machine with 7zip there's an extra folder between the gz and tar, still worked), numpy (no *.egg-info folder), mypy (has empty lines and [extras]) and pywin32 (no Source Distribution)

[Avasam]

Tests found at least one case where this errors out. I'll look into it. Edit: empty lines and [extras] definitions. Now accounted for and fixed.

[Avasam]

Should I delete the files afterward or does it not matter as code meant for the CI? (which I assume just cleans itself up)

[AlexWaygood]

At least some of the test failures will be fixed by https://github.com/typeshed-internal/stub_uploader/pull/95

[Avasam]

I was thinking integration tests might be too fragile, but I can just pin the version I'm testing against. I'm sure you'd like me to add tests for this?

Quick testing script I've been using:

from __future__ import annotations

import requests
from packaging.requirements import Requirement

from stub_uploader.metadata import extract_sdist_requires, validate_response

for req_name in ("pycocotools", "click-default-group", "numpy", "mypy", "pywin32"):  # Pin these
    req = Requirement(req_name)

    resp = requests.get(f"https://pypi.org/pypi/{req.name}/json")
    validate_response(resp, req)
    data = resp.json()

    sdist_data = next(
        (
            url_data
            for url_data in reversed(data["urls"])
            if url_data["packagetype"] == "sdist"
        ),
        None,
    )
    if sdist_data is None:
        print(f"{req_name}: sdist_data is None!")
        continue

    requirements = extract_sdist_requires(sdist_data, req)
    print(f"{req_name}: {list(requirements)}")

Results:

pycocotools: [<Requirement('matplotlib>=2.1.0')>, <Requirement('numpy')>]
click-default-group: [<Requirement('click')>]
numpy: []
mypy: [<Requirement('typing_extensions>=3.10')>, <Requirement('mypy_extensions>=1.0.0')>, <Requirement('tomli>=1.1.0')>, <Requirement('typed_ast<2,>=1.4.0')>, <Requirement('psutil>=4.0')>, <Requirement('pip')>, <Requirement('typed_ast<2,>=1.4.0')>, <Requirement('lxml')>]
pywin32: sdist_data is None!