typeshed-internal / stub_uploader

Scripts and actions to auto-upload typeshed stubs to PyPI
Apache License 2.0
21 stars 16 forks source link

Allow external dependencies #61

Closed hauntsaninja closed 1 year ago

hauntsaninja commented 1 year ago

Resolves https://github.com/python/typeshed/issues/5768 (on stub_uploader side)

In https://github.com/python/typeshed/issues/5768#issuecomment-1251375258 consensus was seemingly reached on Akuli's idea, that external dependencies for stub packages are fine, as long as we validate that the external dependency is a dependency of the upstream package.

The most important part of the implementation is the validation we perform. This happens in verify_typeshed_req and verify_external_req. To help lock things down, Metadata does not expose access to elements of requires without validation.

We rely on PyPI's API to find the dependencies of the upstream. I believe this might not work if our stub has external dependencies and the upstream does not publish wheels. This is not the case currently (as proven by test), does not seem too likely, and it's unclear how to do safely, so will leave as a TODO for the future.

We use uploaded_packages.txt as the source of truth for what is a typeshed dependency. This is important to avoid potential badness around addition and deletion, for instance, if something is added to typeshed, but the distribution is created by someone else before stub_uploader uploads it.

The other set of changes is that I delete most of the graph code that existed previously. The graph code was added in https://github.com/typeshed-internal/stub_uploader/pull/1 and was previously load bearing for security. The idea being to ensure that the graph of transitive dependencies was fully contained within typeshed. This is no longer the case, so we can remove most of it.

I still have some graph code in here, but it's no longer load bearing for security. I keep it around to better preserve uploading semantics, since it seems like it could matter in some edge case scenarios (such as multiple packages being uploaded for the first time that depend on each other). Since we don't have custom needs, we can get away with using the new-ish graphlib from stdlib.

While the graph code has been removed, note that we do still run validation on transitive dependencies for each package. This is accomplished by recursive_verify. (I think the non-transitive validation is probably sufficient, but running this before uploading each package can't hurt)

I added some special-casing for types-gdb. As Akuli pointed out, this avoids us accidentally trusting a gdb package on PyPI if ever someone attempts to add external dependencies to types-gdb.

New code paths have tests. I audited test coverage to make sure of this.

hauntsaninja commented 1 year ago

Bump :-)

ilevkivskyi commented 1 year ago

@hauntsaninja Hm, it looks like one of added tests is broken on master, could you please check?