Comments by Marco from Overcast about Podping

[brianoflondon]

I asked Marco from Overcast about Podping and he gave a thoughtful answer. I thought it worth recording here:

marco:

@Brian of London I think Podping is an interesting idea, but I’m going to pass on it for now. I’ve been crawling over a million podcast feeds regularly for about 8 years, and I’ve learned a lot along the way. First, I’ve learned to do it very efficiently for both my servers and everyone else’s that I’m crawling. You’d be surprised how few servers I use in this role.

Second, we already have ancient, standard HTTP caching mechanisms in place (If-None-Match, If-Modified-Since) that could save tons of resources on feed-publishers’ servers if they implemented them correctly — but they don’t. A shocking number of feed publishers don’t implement HTTP caching at all, and many publishers implement it incorrectly, sending 304s for hours after the feed has, in fact, been modified.

Therefore — third — I can’t rely on any standard or spec telling me that a feed has not been updated for very long. Whatever time interval would cause someone to complain to me that they’re not getting the new episode of a podcast yet — maybe 30 minutes — I need to ignore caching headers, ping TTLs, etc. and recrawl the feed anyway at that frequency, at least.

So feed-crawling, for an app as big as mine, needs to be handled completely in-house. I’ve learned, over and over again, that I can’t trust anyone else’s servers or platforms to work well enough to meet my customers’ standards for reliability and timely delivery of new episodes.

Finally, I’ve learned that many “standards” come and go in podcasting, and my time is not well-spent trying to comply with most of them.

Anyone can publish a spec and call it a standard, and a portion of their audience will go harass indie-podcast-app makers and leave 1-star reviews to badger us into supporting it. (I assure you, this happens. Pressure to adopt “Podcasting 2.0" has not been kind to my reviews, and frankly, I’m quite put off by some of the presumption therein.)

If there’s significant inertia behind something, or I think it’s a really good idea with clear benefits to listeners and no significant downsides, I’ll look at it. (Some of the Podcasting 2.0 ideas fit this criteria, and I’m working on support for some of them — but I’ll do it on my own timeline, balancing it with other priorities.)

But only a small fraction of proposed “standards” actually become standard, and I need to be very careful what I invest my time in.

[brianoflondon]

For the record, this is what he was responding to:

Hey there… first my creds: Overcast has been my primary podcast listening app on my iPhone for as long as I can remember and I’ve been listening to podcasts since I got the 2nd gen iPod with the touch scroll wheel back in the 00's.

I’m heavily involved with Podcasting 2.0 these days which means I have half a dozen other podcast apps primarily the value 4 value features which I full understand @marco feeling they might not be ready for Overcast.

The one thing I had a big hand in though is the Podping new episode notification system. This is a very easy to use and free global notification system for telling the world when a podcast has updated. For Overcast, running an index, this would mean by watching the stream of podpings you could cut out repetitive polling of many RSS feeds. It was seeing how Overcast is polling Podnews’s feed every 4 minutes that brought me here! James Cridland’s Podnews feed puts out an instant notification on Podping every time it changes. You could pick that up in around 45s every single time and never have to check his rss feed again.

Podping has none of the onerous re-subscription overheads of WebSub.

The same applies to all the hosts using Podping (7 at present and a few independent feeds too).

You don’t have to rely on this, but you could try scaling back repetitive polling of shows that use Podping.

The point here is that if Overcast announces that it looks at Podping, the impetus is there for bigger hosting companies to look at sending out the pings. Everyone gets faster notification of new episodes, tons of servers get slowly decommissioned from repetitive polling and global warming is averted and we all live happily every after!

[seakintruth]

Maybe let this sit for a month or so, then reply with some reliability numbers. And emphasize the benefit of speed, and minimal deployment cost in effort and hardware.

I'm running the python to postgresql database / webserver that runs shiny.podping-stats.com off a $600 dollar 2020 desktop from my garage.

One great thing about using a block chain is it's auditability. I can show the maximum time in seconds between two podpings never exceeded x number of seconds, since the last outage back in July.

Still working on some descriptive statistics of podping this week.

[seakintruth]

I understand that dev time is precious, I wonder if he'd be open to setting up a test to run a podping watcher in parallel to his current setup for a week and see how often his current setup would poll faster than the podping watcher, from participating hosts.

I'd be willing to donate to him $100 for him to run the experiment.

I'd wager another $100 that the podping watcher will update faster 98% of the time.

[agates]

I certainly understand why one might not be interested in it just to reduce polling. However, I think, especially with the upcoming version, the following points are be essential in addition to what @seakintruth said:

1. New feed discovery. Especially as adoption grows, this is already a way to announce and find new feeds without depending on "submission" to various different platforms and APIs.
2. Live stream notifications. Personally, as awesome as these are, I think we're ahead of the curve. Once the trend catches on, we'll be ready. May or may not suit Overcast, and that's fine.
3. "Trust." You don't have to trust anyone. You can run your own servers to contribute to Hive (or, I suppose, any other potential future implementation of podping). This is a protocol, not a platform. And as much as I don't care for the social aspects of Hive, the reputation and account system does allow us to help work around abuse/spam/malicious intent without being a controlling actor.

[daveajones]

Feed discovery and public notifications really make this a thing that hasn’t existed before. Those two things alone are things websub could never provide.