More useful pagination

[jameysharp]

I'd like to see the JSON Feed spec take some lessons from RFC5005, the "Feed Paging and Archiving" spec for Atom and RSS.

As it stands, JSON Feed's next_url attribute provides part of RFC5005's section 3 ("Paged Feeds"), but could be extended to allow first/prev/last navigation. I'm not personally particularly interested in this case, but it is frequently the easiest to retrofit into existing content management systems.

The absence of a next_url does not say anything about whether there are older entries in the feed, so it'd be useful to allow a flag like RFC5005's section 2 ("Complete Feeds") indicating that no entries exist besides those in the current document.

And RFC5005 section 4 ("Archived Feeds") provides for feed pages which clients may cache indefinitely. Feeds which follow the rules in that section enable readers to reliably mirror any amount of history from the feed without ever polling for updates on any page besides the current one. (And of course even that polling can be optimized with WebSub.)

Each of these is useful for different cases and I'd encourage providing all of them, but especially sections 2 and 4. The knowledge that all entries can be retrieved again from the origin feed has two immediate benefits for feed readers which keep local copies of old posts:

• They can delete local copies of posts which have been removed upstream.
• They can treat local copies purely as a cache which can be dumped as needed, regardless of how much history the user has requested to have preserved, allowing the illusion that the full archive is available locally.

For archived feeds, tombstones like those specified in RFC6721, 'The Atom "deleted-entry" Element', are also useful.

[manton]

Thanks @jameysharp. How would you feel if we started by adding prev_url first and then consider other options later? For JSON Feed 1.1 I'd love to keep it to the bare minimum changes since it's supposed to be a very minor update.

[jameysharp]

Now that 1.1 is out, maybe we can revisit this.

While it's nice to be able to page through the feed in both directions, I'm not particularly concerned about that. I'm more concerned about readers' ability to cache older parts of the feed and efficiently detect if they've changed.

It occurs to me that most of what I want could be delegated to other specifications if JSON Feed offered a new property, both at top-level and inside items, compatible with RFC8288, "Web Linking". It would be equivalent to <atom:link> tags, HTML's <link rel> and <a rel> tags, and the HTTP Link header. I'd like the spec to link to the IANA Link Relations registry and it'd be nice if it also mentioned specifications of particular relevance for feeds, including RFC5005.

If you adopt that suggestion, the only RFC5005-like feature I'd especially want to add to JSON Feed itself is this top-level boolean property:

• complete: The entries in this document are a complete record of the entries which should be displayed for this feed. If the feed reader has saved posts which no longer appear here, it SHOULD remove them.

I'd kind of like to also specify this top-level boolean property, as it's helpful for detecting if the right feed URL is being used:

• archive: This document does not contain the most recent posts for this feed. In this case, the publisher SHOULD also provide the equivalent of <atom:link rel="current"> to point to the feed containing the most recent posts.

[Guichaguri]

+1 on this. The prev_url would be really useful.

Also, I would love to know what purpose does feed_url have? Is it the current feed URL or the first page?

[dshanske]

I decided I want to add next_url to the WordPress JSONFeed plugin, as it would be trivial to do so, as WordPress feeds support pagination. I think prev_url makes sense in that regard.

@Guichaguri I would interpret it as the start of the feed, related to home_page_url, which is the HTML version of the feed. So, we don't need a first.

With the archive properties, is anyone consuming them from RSS and Atom?

[jameysharp]

I've built various producers and consumers of RFC5005 section 4 metadata, such as https://github.com/jameysharp/crawl-rss. I've seen other folks use RFC5005 section 3, which corresponds to the next_url/prev_url style of links, but doesn't support efficient caching. For more context, I wrote about why I care a while back in "Feed reader cache coherency with JMAP and RFC5005".

By the way, for WordPress specifically, I've written both a plugin and a proxy that add this metadata to the WordPress RSS and Atom feeds.

[dshanske]

@jameysharp Will have a look at those. Just wondering who is using them.