search

leafac / kill-the-newsletter

Convert email newsletters into Atom feeds
https://kill-the-newsletter.com
MIT License
2.31k stars 113 forks source link

Feature Request: PubSubHubbub/WebSub support #68

Closed mrnoname1000 closed 3 weeks ago

mrnoname1000 commented 1 year ago

Given the nature of email, being able to push articles to clients the moment they are received would be convenient. This can be achieved by implementing PubSubHubbub/WebSub to push to clients that are publicly available (i.e. web-based or client/server feed readers). This would probably reduce bandwidth usage too, as frequent refreshes would turn into a single request per new email.

The PubSubHubbub Github organization has lots of good resources on the protocol.

leafac commented 1 year ago

Thanks for the information. That sounds interesting. I’ll look into it…

kevincox commented 1 year ago

As well as real-time updates being great for users this could reduce load on the service. I run a small reader but with a couple of subscriptions I end up polling every few minutes. I'm sure for larger reader services this adds up very quickly.

When WebSub is supported my service switches to only polling each feed once a week (plus a WebSub subscription request). So in exchange for a small amount of storage to remember subscriptions you can save a lot of CPU and network load. I don't know if that is a notable concern right now but using less resources is always nice.

The protocol is also fairly simple. Basically:

  1. Advertise support in the feed. (Add a <link rel=hub .../> tag)
  2. Accept subscription requests. (HTTP GET)
  3. Validation subscription requests. (Send HTTP GET)
  4. Send out new updates (Send HTTP post)
edsimpsons83 commented 11 months ago

This would be an amazing feature to add. I would be more then happy to donate to a bounty source or something similar as well to get this going.

leafac commented 3 weeks ago

Hi y’all,

I implemented WebSub support as part of 2.0.4.

Thanks for bringing this to my attention—I didn’t even know WebSub was a thing and it was fun to implement 🤗

Enjoy.

@edsimpsons83:

This would be an amazing feature to add. I would be more then happy to donate to a bounty source or something similar as well to get this going.

Oh, that’s very nice of you. Thanks. Here’s how you can support me:

Patreon · PayPal · GitHub Sponsors

mrnoname1000 commented 3 weeks ago

Thanks for adding support! I can see the subscription in the FreshRSS UI and logs, but KTN doesn't appear to be pushing new entries. I've tested with other feeds and the reader does seem to be working properly, though it wasn't before I checked :P

Is it working for anyone else using a different feed reader?

leafac commented 3 weeks ago

I've tested with https://websub.rocks/hub when I was developing the feature and it worked.

kevincox commented 3 weeks ago

Thanks for implementing this. One odd thing that I noticed is that subscription validation takes a while. I've seen 1h, I've seen 5h I've seen over a day (or maybe there was just an outage with no verification sent in that time).

The WebSub specification Section 5.1.2 says "The hub SHOULD perform the verification and validation of intent as soon as possible". These times don't seem "as soon as possible".

This is just a "SHOULD" but what ends up happening with my reader is that it assumes that the subscription failed for some reason so it creates a new one on the next feed check. Then it ends up getting a handful of subscription validations in a row.

I haven't actually seen any entries posted yet, but that seems to be just that the feeds my readers are using are quiet. I'm adding my own feed and I'll do a test as soon as the subscription validation occurs.

leafac commented 3 weeks ago

Hi @kevincox,

Thanks for reporting.

The issue that there are too many verifications/updates to send and the server isn’t keeping up.

I’m surprised by how popular WebSub turned out to be. But that’s a great problem to have 🤗

I’ll investigate what I can do about it…

kevincox commented 2 weeks ago

I'd just like to pop in and say that once the subscription is eventually confirmed it does work properly. I sent a message to my test feed and got a pushed feed update quite quickly. 🎉

mrnoname1000 commented 2 weeks ago

Seems like @kevincox is right, the confirmation does take some time but afterwards websub works great! I also sent an email to my test feed and it arrived in my feed reader within seconds.

leafac commented 1 week ago

I tweaked some settings and now the background job works seem to be managing the load. Please let me know if things are speedy for y’all now.

kevincox commented 1 week ago

As of about 2024-06-23 16:00 UTC all of the subscriptions have been confirmed within a second. But my service only has a few feeds so not a huge sample point. I'll try to remember to check again in a week after they refresh a few times (as they seem to be 24h leases) and confirm that it is stable.