Explore building third party stubs as packages

srittau commented 6 years ago

This is an alternative to #2440 (disallowing third-party stubs). The idea is that typeshed remains/becomes a central repository for third-party stubs that are not bundled with the parent package, similar to DefinitelyTyped. In the future I expect type checkers will not want to bundle all third-party stubs for a variety of reasons, so third-party stubs would be distributed as separate PEP 561 stub-only packages, one per upstream package.

(I tried to integrate points raised there into this issue, especially those by @JukkaL in this comment.)

Advantages

Due to typeshed's tests, packages in typeshed will continue to work with the latests versions of mypy and pytype.
Basic level of consistency and standards due to review by typeshed maintainers.
Consistent naming scheme for third-party stubs packages, allowing users to just "pip install <guessed name>" and it will work when there are stubs.
Tooling (like tests) is easier to manage as it can remain part of the typeshed package.
Easier to contribute to stubs, since contributors don't need to learn the intricacies of multiple stubs projects.
No need to start a separate project just to distribute stubs for a new package.

Issues

Workload issues for typeshed maintainers.
Typeshed maintainers will often not be familiar with the package for which pull requests are opened.
Publishing stubs takes longer due to the necessary reviews.

Further Considerations

What should the generated packages be called? @ethanhs's PEP 561 actually requires stubs-only package to be named <package>-stubs. typeshed could squat these names and release them (and remove the stubs) on the request of upstream maintainers. Alternatively, typeshed could add a common prefix or suffix (ts, typeshed) or in addition to or instead of the -stubs suffix. This would be in violation of PEP 561, so we'd need to get broader consensus to amend the PEP. My personal favorite would be <package>-ts.

To guarantee a fairly quick turnaround on stubs, to minimize work for publishing stubs, and to prevent all third-party stub packages to be updated whenever a new typeshed version is released, stubs for a specific third-party package should be published automatically when it changes.

Possible Implementation

Add a generic setup-tp.py to typeshed that takes its package name from the directory it's in and uses the current date and time as version number.
Amend the CI process for master only so that after a successful test run, for every third-party package that was changed since the last successful run, the following is done automatically:
1. Copy setup-tp.py into the third-party module directory as setup.py.
2. Build the package in that directory.
3. Upload to pypi.

ilevkivskyi commented 3 years ago

Nice, thanks! I will make a PR soon.

ilevkivskyi commented 3 years ago

OK, I merged the big PR with directory re-structure. I will continue working on typeshed -> PyPI auto-upload later this evening.

ilevkivskyi commented 3 years ago

Oh, I forgot to update README.md and CONTRIBUTING.md they still reference some things from old directory structure (and typeshed being a submodule in mypy). I will make a follow-up PR later unless someone beats me to it.

ilevkivskyi commented 3 years ago

In the meantime, I have manually tested PyPI upload GitHub actions, they seem to work. After @JukkaL double-checks the code tomorrow morning, I will switch the cron diff upload action from --dry-run to normal. Then it should start auto-uploading wheels for modified third-party stubs every three hours. I will then make a trivial PR to check the integration end-to-end.

ilevkivskyi commented 3 years ago

Thanks @rchen152 for updating the pytype test! Now we only need to fix stubtest and mypy_primer, cc @hauntsaninja.

erictraut commented 3 years ago

Thanks @ilevkivskyi and @srittau for making this change. I just updated the import logic in pyright, and it now works with the new typeshed directory structure.

ilevkivskyi commented 3 years ago

I think we are almost ready, sorry for a little delay, we were not able to enable the auto-upload today, we will enable it Monday morning. In the meantime, I think it is already OK to start merging PRs again.

ymotongpoo commented 3 years ago

First of all, thanks for working for the better type hinting. I look forward to all type information packages coming out to PyPI. Now I have a question on the upcoming updates on the type information metadata in all libraries in this repository.

How does the metadata file (METADATA.toml) will include the supporting runtime package versions? I saw a discussion around the versioning of stub packages but wasn't confirmed the concrete mention on the description on the metadata file.

If I missed the exact part that tells the answer for them, apologies in advance.

ilevkivskyi commented 3 years ago

I finally switched the auto-upload task from --dry-run to normal https://github.com/typeshed-internal/stub_uploader/commit/96cb137b2e94af55261ada4cc97365171c9b4952, so I think this issue may now be closed as implemented.

ilevkivskyi commented 3 years ago

How does the metadata file (METADATA.toml) will include the supporting runtime package versions?

Currently you can include version = "x.y" in metadata which means that this stub package only valid for runtime package version x.y or newer.

ymotongpoo commented 3 years ago

@ilevkivskyi Thank you for the reply. Then now the stub packages will be automatically uploaded to PyPI based on that version information?

For example, the stub for paramiko is version "0.1", whose runtime package's current latest version is "2.7.2". In this example, given paramiko is following semver, 2 major version diff means huge interface breaking changes happened during the update, which means at least the stub version should be 2.0, preferably 2.7. And I see other similar cases.

Can we assume that now all stub packages are ready to accept the pull requests for such version number corrections?

hauntsaninja commented 3 years ago

Yes, we'd accept PRs that update METADATA.toml to the oldest version of the package that the stubs reflect support for (#4981)

JukkaL commented 3 years ago

When adding new stubs, the version should reflect the actual version of the package that stubs were created for. Since this will be hard to determine reliably for existing stubs, it's okay for the version to refer to some older version with which they are still compatible.

We probably also don't want the version to refer to a more recent version version of the package, or a version that is poorly supported. If a stub is created for version 2.3 of a library but also contains a few 2.4 features (but we know/suspect that some important 2.4 features are missing), I think it's fine to continue to use 2.3 as the version of the stubs.

ymotongpoo commented 3 years ago

@hauntsaninja @JukkaL Thank you for the reply.

I understand that we need to avoid the confusion that the version changes will bring and we can keep using versions in the existing METADATA.toml files. However, that applies only to the case the stub version is in the same major version as the package current, and not to the case I referred to. (i.e. 2 major version diffs)

Anyway, it's great that the discussion is already happening in #4981 and I'm happy to contribute to this part.

ofek commented 2 months ago

FYI I just opened a PEP about package repository namespaces and I used typeshed as the first motivating example 🙂 https://discuss.python.org/t/pep-752-package-repository-namespaces/61227

python / typeshed