Support for WebSub

[Alkarex]

Hello, This is a suggestion to consider implementing WebSub / PubSubHubbub for the existing RSS/Atom feeds.

(Already mentioned in https://github.com/Chocobozzz/PeerTube/issues/493#issuecomment-386458200 )

---

Roadmap

• [x] launch a notification task every time the ressource is updated with new content/modifications
• [x] add the Hub to the feed (it's already supported by pfeed as a parameter but we might want to add multiple hub support)

[rigelk]

@Alkarex do you know of any potential clients that would effectively use our WebSub feeds? (apart from the internal notification system)

[Alkarex]

My use-case is for RSS/Atom feed readers. For instance, I am one of the developers of FreshRSS, but some larger actors include Feedly, The Old Reader, Inoreader, NewsBlur, IFTTT, etc.

P.S.: While RSS/Atom works fine with pull, WebSub / PubSubHubbub enables push, which is faster and often times more resource efficient. For instance, FreshRSS automatically reduces the pull frequency to at max one per day when a feed supports WebSub / PubSubHubbub.

[Alkarex]

Maybe @julien51 could have a tip there

[rigelk]

@Alkarex @julien51 do you think we should setup a WebSub hub ourselves, as common infrastructure for PeerTube instances?

[rigelk]

ping @Alkarex @julien51 - before I go forward with the PR, I need to know if pubsubhubbub.superfeedr.com is the only hub I should care about for know. Being new to WebSub, I don't see a lot of hubs and this one seems like the default to stand out.

In your experience, is it fine if we default to that hub, or should we use/setup another one?

[Alkarex]

Hello @rigelk Here are some statistics from my FreshRSS instance:

$ find -name '!hub.json' | xargs jq -r .hub | sort | uniq -c | sort -nr
     38 http://pubsubhubbub.appspot.com/
      2 https://pubsubhubbub.appspot.com
      2 https://mamot.fr/api/push
      1 https://websub.rocks/blog/102/DZ0RbPbGlPdQVR6xPX8U/hub
      1 https://techcrunch.com/?pushpress=hub
      1 https://pubsubhubbub.superfeedr.com/
      1 https://kristinahobbs.com/?pushpress=hub
      1 https://humblr.social/api/push
      1 https://henningjust.wordpress.com/?pushpress=hub
      1 https://hackaday.com/?pushpress=hub
      1 https://f.a80.fr/pubsubhubbub/alkarex
      1 https://blog.ouseful.info/?pushpress=hub
      1 http://pubsubhubbub.superfeedr.com/
      1 http://medium.superfeedr.com/
      1 http://hackaday.com/?pushpress=hub
      1 http://feedpress.superfeedr.com/

When using a third-party hub, https://pubsubhubbub.appspot.com (Google) or some variants is the most used (especially because it is used by https://feedburner.google.com and other Google services), but https://pubsubhubbub.superfeedr.com is also a good option (used by e.g. Medium http://medium.superfeedr.com).

But instead of a third-party hub, some systems are using a built-in mechanism. This is the case for WordPress (when using a plug-in https://wordpress.org/plugins/pubsubhubbub/ or when hosted on WordPress.com), Friendica, GNU Social...

Using a third-party hub is easier (it is really just sending a very simple HTTP request when something has changed) but less decentralised and less powerful. In particular, it is not practical to use a third-party hub if you would like to support custom filters (e.g. having clients interested only in some specific news but not all).

Using a shared hub as a common infrastructure for PeerTube instances would be half way between those two approaches.

I can gladly help with testing / architecture / and other feedback.

For testing, see https://websub.rocks/publisher and https://websub.rocks/subscriber (the subscription part might not be relevant for you).

Here are some examples of feeds using WebSub with various mechanisms:

• https://push-pub.appspot.com/ (Best one for testing)
• http://rss.slashdot.org/slashdot/slashdotMain
• http://feeds.arstechnica.com/arstechnica/index
• https://hackaday.com/feed/
• https://feeds.feedburner.com/techcrunch and other feeds served via Feedburner (many)
• https://www.blogger.com/feeds/2471378914199150966/posts/default and other Blogger sites
• WordPress sites hosted on https://wordpress.com or using https://wordpress.org/plugins/pubsubhubbub/
• Friendica instances such as https://f.a80.fr/feed/alkarex/
• Etc.

Let me know whether that answers your questions so far :-)

[Alkarex]

P.S.: Maybe https://github.com/hemerajs/websub-hub could be inspiring (not tried)

[kevincox]

A hub is pretty easy to implement. It just needs to track subscriptions and notify subscribers when there is a new video. It is very similar to the ActivityPub route really, just a different format. For example WordPress has the PuSHPress Plugin which is widely used and installed by default on all wordpress.com instances. This may be an attractive option since all PeerTube instances have a service running already so implementing a hub would avoid needing a separate dependency.

But if the initial implementation would go faster with public hub I don't see a major downside. Especially if it is configurable. (In practice public hubs use the same publish notification API)