RSS feed generation, workflow, pipeline

[gerbrent]

The BIG question: how will the RSS feed generation pipeline function, in a best-case scenario?

from devotoare:

Is the plan for the rss feed for Hugo to generate the new feeds or do we need to build a server for that? Not sure if Hugo can do that... I assume given that it can do JSON, it can.

[kbondarev]

Here's my understanding of the future publishing process. I'm not entirely sure whether that's the plan or not, so please correct my if I'm wrong:

• the Hugo markdown files for all show episodes are source of truth. Meaning when a new episode is about to be published, such file would be created manually and all the variables in it would be set (maybe latter on via a front matter plugin on vs code which would make a GUI form for it)
• Then based on that we generate the whole JB website, the websites for each specific show, and all the RSS feeds.
• As a consequence Fireside websites and their RSS would become obsolete

---

Now, taking all that in mind here's the answer for your question:

Hugo can generate basically any custom output format. What would we need to build is the template for the rss xml file using the variables from the JSON front-matter of each episode. It's very similar now the HTML pages are created using the templates from the JB theme folder, but inside it would be XML.

The current json (in which all the data created by the scraper) might be missing some things that we would need for the RSS. Will need to take a closer look at the current RSS of the shows, the new standards from podcasting index, and add anything that might be missing. I'll go over it tomorrow.

[StefanS-O]

We do generate RSS / Podcatcher compatible feeds already:

All Shows: https://jupiterbroadcasting.net/show/index.xml Specific shows, e.g. Coder Radio: https://jupiterbroadcasting.net/show/coder-radio/index.xml

The template file for this is here: https://github.com/JupiterBroadcasting/jupiterbroadcasting.com/blob/main/themes/jb/layouts/section/list.rss.xml

[kbondarev]

I didn't even know we had that already implemented. That's great!

I think we do need to modify the template a little to include a bunch of those new tags from Podcastingindex 2.0 namespace

---

This might need to be a separate issue, but somewhat related here: Live items in that new spec might be somehow integrated with the JB Calendar. Maybe each time before Hugo publishes it should also look at next 7 days of the calendar and based on that schedule those live items with a "pending" status. And then I assume JBOT would be responsible for switching the status to "live" and "ended" (perhaps related to this issue #41)

[gerbrent]

Here's what we discussed and decided for now internally at JB regarding RSS feeds for JB.com 1.0:

• Fireside will continue to manage the feeds, as is currently the case, thus preserving the back-end show publishing pipeline.
• We should remind ourselves: The priority is, at this point and on this first iteration of the hugo site, to replace the functionality of the current main JB Wordpress site only. Everything else is bonus, including having Hugo manage RSS. Thus, feeds are not currently in the scope of the Hugo site.
• The RSS and publishing pipeline can certainly be evolved later, but is not a requirement for the first go-to-live of the Hugo site here, for a great variety of intricate and esoteric reasons.

So! The above are all great ideas, and will certainly be revisited in time. I understand this challenges the "hugo as source-of-truth" principle... no small thing.

[ChrisLAS]

What Brent said. But I figured I'd share a bit more background with you all. I'm going to try and talk about this in Office Hours 8 a bit for those of you who want more info/context than I can put in text, but I wanted to post our current plans for RSS generation.

RSS feed generation for the podcasts is a bit more complicated than it might look at first pass.

Each show has quite a few feeds, each with various end-use cases in mind.

Take LUP for example:

• A main MP3 RSS feed (The "public" feed)
• A members ad free feed
• A members live stream feed.
• One entry in the All Show Audio feed
• One entry in the All Shows Video feed

👆 That's just one show off the top of my head. And not even the most complicated.

And the features those feeds support IE Podcasting 2.0 namespaces are still in flux right now. We are going to be adding more and more P2.0 feature support this year and next.

Additionally, things get even more complex once we start using the Podcasting 2.0 live tag. Which requires an update to the show's RSS feed (and the all shows feeds) when:

• You're about to go live
• When you go live
• When you're no longer live

👆 Each one of those will require whichever Podcast is going live to have its feed updated 3 times, without any episode, or web page ever being updated.

We also want the flexibility in the future of being able to offer meta-feeds, ie, a "Linux only feed" or "interviews feed", etc.

So while we could generate one (or maybe more) of the feeds using the website, it would just add fragmentation to how we build the RSS feeds and not functionally ideal for JB.

I could TOTALLY see it working in some setups. IE, if we had a simpler lineup, fewer shows/feeds, etc.

[elreydetoda]

Ya...wow...while really informative (thank you for the background (I love hearing this type of info, because it helps drive the technical solution) 😁) , that's a lot 😅

There's a couple ( or at least one) that I can think of off hand, but I'll post it after I finish work today (to let the info percolate and validate I'm understanding things properly).

About this part (note for myself):

Each one of those will require whichever Podcast is going live to have its feed updated 3 times, without any episode, or web page ever being updated.

My first thoughts is that it should be a python script in a separate repo like how the scraper is.

[elreydetoda]

So, @gerbrent meta question for this issue...should this issue still be a 1.0 task?

Should we create another issue for 1.0 saying point to fireside for the RSS feeds, or keep it all here?

[gerbrent]

good point, it was in the milestone for decision making, lets make a new issue for the implementation.