API for fetching latest and stable versions

techtonik commented 6 years ago

What's the problem this feature will solve?

Tools could use simple lookup logic for checking their latest and stable versions. The way it is currently done in pip is crazy:

https://github.com/pypa/pip/blob/35d51f1f405f6f31d68c800e73307686ec029254/src/pip/_internal/utils/outdated.py#L104-L118

Describe the solution you'd like

It should be as simple as fetching https://pypi.org/pypi/pip/latest/json or https://pypi.org/pypi/pip/stable/json endpoints.

bskinn commented 4 years ago

I don't know what fraction of PyPI's bandwidth is consumed by JSON requests, but I feel a bit bad pulling down the entire /pypi/{pkg}/json endpoint for a project/package when all I want is to check the most recent version number.

di commented 4 years ago

@bskinn Most of the bandwidth is consumed by distributions (files), not JSON. And even then, 90-something percent of it is cached by our CDN -- so don't feel bad 🙂

That said, this API makes sense to me and would be relatively easy to implement. The hard part is getting it right.

Something like the following seems to cover all use cases to me:

/latest/json Latest non-prerelease if available, otherwise the latest prerelease
/latest-stable/json Latest non-prerelease
/latest-unstable/json Latest version including prereleases

Should just be redirects to the actual versions? E.g. /pypi/pip/latest/json -> /pypi/pip/20.1.1/json?

bskinn commented 4 years ago

<nod>, those three, and their names, make sense to me.

Main catch scenarios are (1) for all three, if no versions are available, and (2) specifically for latest-stable, in the case that no non-prerelease versions are available?

di commented 4 years ago

I imagine that they'd be 404 instead in that case, same as if the project didn't actually exist.

Another question, should any of these include yanked releases? I think not.

bskinn commented 4 years ago

Yeah, I figured they'd 404. Not knowing the warehouse codebase that well, though, I wasn't sure if there's already machinery there that would cleanly handle this particular failure mode & kick out the 404.

Agreed, I figure it should mimic the behavior of pip in dealing with yanked projects. Since this API call is intrinsically an operation where the user can't indicate a specific release, yanked releases should never be returned.

bskinn commented 4 years ago

If I read the model code correctly, Project.latest_version would supply the correct version for /latest/json/, including masking-out of any yanked releases.

If so (and if this is the right place in the codebase to implement the search), could just implement two variations that provide latest-stable and latest-unstable?

di commented 4 years ago

That would make sense to me!

bskinn commented 4 years ago

Ok... I've been poking at this, and I think I'll try to put together a PR for it.

One question, @di -- what is the semantic meaning of a NULL value for Release.is_prerelease?

Project.latest_version uses .nullslast(), which seems to suggest that there may be some edge cases where returning such a Release is desired. But, I'm wondering whether a new Project.latest_stable_version should also use .nullslast() in the same way, or if it should exclude .is_prerelease NULLs from the query altogether.

Regardless of the above, I figure Project.latest_unstable_version should follow .latest_version and also use .nullslast().

di commented 4 years ago

As far as I can tell, it should never actually return NULL. It's a column property:

https://github.com/pypa/warehouse/blob/bd2b3a22f4f84072eb03abb771a88ddab1489e7a/warehouse/packaging/models.py#L357

defined by this function:

https://github.com/pypa/warehouse/blob/1fbb4ac752e68b5840b9e09b68e44a165569bfa6/warehouse/migrations/versions/e7b09b5c089d_add_pep440_is_prerelease.py#L29-L34

but Release.version can't be NULL:

https://github.com/pypa/warehouse/blob/bd2b3a22f4f84072eb03abb771a88ddab1489e7a/warehouse/packaging/models.py#L355

di commented 4 years ago

Looks like it was added in https://github.com/pypa/warehouse/pull/3470/

bskinn commented 4 years ago

Maybe it's a guard against a mangled table? Regardless, looks like there's no need to consider the NULL case; and shouldn't hurt anything to just keep the .nullslast() in the call flow, in case. Thanks!

graingert commented 4 years ago

I'd also like to be able to link to the latest and stable versions:

.. image:: https://img.shields.io/pypi/v/modernize/latest?logo=pypi
    :alt: PyPI Latest
    :target: https://pypi.org/project/modernize/latest
.. image:: https://img.shields.io/pypi/v/modernize/stable?logo=pypi
    :alt: PyPI Stable
    :target: https://pypi.org/project/modernize/stable

EwoutH commented 2 months ago

I would still find this useful!

pypi / warehouse

API for fetching latest and stable versions #4663