Podcast Block :: Not Showing Newest Podcast

[sbathompson-he]

Steps to reproduce the behavior

1. Go to the Users homepage but the newest podcast is not showing
2. Click on the Editor and it shows the newest podcast / Part-2
3. Readding the RSS feed, deleting the block and adding the feed or even creating a copy of the page did not resolve the issue
4. Here is a video of the issue

https://user-images.githubusercontent.com/18581859/173178831-e926f5e6-f017-46a2-8d52-5488a1370c7c.mov

What I expected to happen

• The next podcast should pop up on the homepage

What actually happened

• The last podcast is showing

24319598-hc 3791418-zen

Workaround

This should but may not always work:

• Atomic sites: enable Clear Cache (see reference)
• Simple sites: clear the site cache through the μ menu

[sourourn]

User came back to follow up again: #28165381-hc

[ivan-ottinger]

Another report: 3846904-zd-woothemes

Even though the RSS feed is correct, it seems to take several hours until the newest episode shows up.

It looks like when we clear the site cache through the μ menu (Simple site), it helps to resolve the issue. Regardless, the user is frustrated that their newest episode displays everywhere else correctly (e.g. iTunes), but is still not updated on their site.

[metabreakr]

24226051-hc / 3967023-zen

[kuropixel]

Another report: 2452046-hc / 4199678-zen

The user (simple site) had a total of 8 podcasts episodes, but only 7 were showing up, but not the latest. On a test simple site, I added a podcast block and input the user's RSS feed with a Podcast block and the latest episodes showed.

I tried clearing the site's cache via directions "clear the site cache through the μ menu (Simple site)" but this did not resolve the user's issue. We also tried new posts, new podcast blocks, but nothing resolved for their site specifically.

[kosiew]

Reviewed above cases. In all of them, the latest podcast issue is appearing in the podcast player block later. Triaging and await developer to confirm this is the behavior of the podcast player block.

[Jaccuse]

Another report here: 31300702-hc Same user as in @kuropixel's comment above https://github.com/Automattic/wp-calypso/issues/50740#issuecomment-893180751

[lunasera41]

Another report here: 30137359-hc

[kerrynicl]

Another report here. 30257991-HC . Clearing site cache resolved issue

[nicolynramos]

Another here: 34495930-hc. Clearing cache resolved

[ariel-maidana]

Another report in 4936393-zd-woothemes (simple site). Clearing the page cache from the μ menu solved the issue.

[Arnodt02]

Another report in 4936393-zd-woothemes (simple site). Clearing the page cache from the μ menu solved the issue.

[formosattic]

+1 4936393-zd-woothemes (posted in https://github.com/Automattic/wp-calypso/issues/62942).

Added workaround in the initial post.

[mxhassani]

Similar case here: 30792949-hc The simple site workaround helped.

[wiesenhauss]

4936393-zd reports the same issue again (Simple site).

[aleone89]

Another report of this here hc-35402620

[rickmgithub]

Same again - 4936393-zd

[rickmgithub]

My own sites. I go into CLI and do a cache flush and it updates.

[cecilearkay]

Another report in 35480479-hc

[formosattic]

Bumping the priority to High as the fix may need to be applied more than once; and for Simple sites, it requires the intervention of an HE which isn't an efficient support process.

@rickmgithub: How frequently are you having to do the manual refresh? Does it happen for each new episode?

[rickmgithub]

@formosattic Hi. I have to do it every time or wait for it to happen eventually.

[yansern]

Cache resets every 1 hour. So if there is a new episode, give it 1 hour.

If user wants it faster, clear the site cache manually.

[mriyazuddin]

• [ ] Another one reported here: 5266213

[rickmgithub]

Another report 5266213-zd User indicated it takes more than 1 hour. Asked user for more details on time scale

[zyousafi]

Another report 5266213-zd User indicated it takes more than 1 hour. Asked user for more details on time scale

The user came back in 5266213-zd and confirmed that it takes anywhere from 6 to 12 hours. Longer than 6 for sure, but not more than 12

[daledupreez]

I've done some additional investigation, and it looks like the root cause is that the default WordPress caching for RSS feeds (which includes podcasts) is for 12 hours as per this code. As far as I can tell, we're not overriding that for podcasts, which matches the experiences noted above, where the cache expiry is anywhere up to 12 hours, depending on when the podcast RSS was last fetched.

There are at least two ways we can improve the behaviour, and a workaround for sites who can run plugins and custom code.

Approach 1: Add a block setting to specify the length of time to cache the podcast feed

With this approach, we would add a setting to the block that allows users to specify a custom cache timeout for that podcast. This is relatively simple and keeps control close to the place the podcast is being added to the page. (From an implementation perspective, we'd add a setting to the block, likely as an integer with a 1 hour minimum, and then use the relevant filter to modify the cache timeout to be that value.)

The downside is that different blocks could specify the same podcast URL, but specify different cache timeouts, which might lead users to become extremely confused about why their podcasts aren't updating when they expect them to. This would also be subject to other fetchers of the podcast RSS feed using the default timeout.

Approach 2: Manage feed timeouts centrally

With this approach, we'd build a central location for managing cache timeouts per feed. This would move management away from the block editor, and is something that might be overly specific (and overkill) for the feature we actually need. This is also something we almost certainly don't want in the Jetpack plugin, and would come with a notable maintenance burden. It's possible a plugin already exists for this, but it's not something we'd want to build.

Workaround if you can add plugins or code

Either by writing custom PHP, or by adding some PHP using a plugin like Code Snippets, you can modify the cache timeout for your podcast feed using code something like the following:

function custom_reduce_podcast_cache_timeout( $timeout_in_seconds, $url ) {
  if ( $url === 'https://example.com/your-podcast-url/' ) {
    return HOUR_IN_SECONDS; // set the cache timeout to be one hour -- we don't want this too short, as we might make too many requests for the podcast, but this means it should take at most an hour to get updated on your website
  }

  return $timeout_in_seconds;
}

add_filter( 'wp_feed_cache_transient_lifetime', 'custom_reduce_podcast_cache_timeout', 10, 2 );

NOTE: I haven't tested the code above, but I think it should be pretty close to what's neeed for this case.

[mpkelly]

Hi @daledupreez, I want to propose a different approach to the two above.

I don't think an arbitrary timeout will ever work well: it won't be very effective if it is too small, and if it's too big, it will lead to more support tickets. I think users expect newly added content to show instantly, which is how most apps work these days. Ideally, our cache would be based around events instead of timeouts, i.e. when new RSS content is added, the cache is automatically recreated to contain the latest content, which remains cached until new content is added.

The part I am unsure about is how difficult this would be with our current codebase - is it worth us doing some analysis? Maybe we could consider a phased migration to events starting with podcasts?

[daledupreez]

I dug into this further yesterday, and I think we can move towards a better default experience by reducing the cache timeout to something like an hour. As per a discussion I had with @mpkelly, we likely need to tackle this as follows:

1. Add instrumentation so we understand how many outbound requests we're currently making for the podcast player, and what responses we're getting back (primarily whether we're getting 2xx responses or 304 responses, but possibly whether we're getting Last-Modified or ETag response headers)
2. Continue work on https://github.com/Automattic/jetpack/pull/24966 so we can conditionally add lower cache values for exploration
3. Once we have a baseline from 1, add filters to reduce the cache timeout so we can monitor the impact on our outbound API calls

[daledupreez]

As a quick update on this, I captured a more detailed plan for this internally in pdP0P1-3w-p2, and I just deployed D84039-code to start gathering stats on the response codes (and volume) we are seeing at the moment.

[renata-franco]

Hi @daledupreez and @mpkelly. Any progress on this issue? Can we get an update, please?

[github-actions[bot]]

Support References

This comment is automatically generated. Please do not edit it.

• [ ] 3791418-zen
• [ ] 3846904-zen
• [ ] 3967023-zen
• [ ] 4199678-zen
• [ ] 4936393-zen
• [ ] 5266213-zen
• [ ] 5381587-zen
• [ ] 5508841-zen
• [ ] 5286257-zen

[daledupreez]

Hi @renata-franco, thanks for pushing us on this issue! Automattic/jetpack#24966 was approved earlier today, so we will be merging that to the trunk branch tomorrow. That should allow us to roll the new capability out to WordPress.com Simple sites tomorrow, with sites that have plugins getting access to the capability early next week.

I have also started work on D84947-code, which will allow us to start rolling out a reduced cache period of 1 hour to a subset of sites. We plan to do that incrementally so we can confirm whether we see a big change in the number of outbound RSS fetches we need to perform. So we should start seeing some movement tomorrow or Friday, and hopefully a broader rollout next week.

Once the feature is available to sites with plugins, I will share a specific recipe for reducing the amount of time that podcasts feeds are cached for.

[renata-franco]

Thanks so much for the update, @daledupreez :D I am so glad that this is moving along. I will wait patiently! 💯 🤩

[daledupreez]

We are in the process of testing a shorter 1 hour cache period for podcast feeds. We started with 10% of WordPress.com Simple sites earlier today, and have just bumped the number up to 50% as there didn't seem to be any major impacts.

For reference, we bumped the roll-out up to 50% in D85010-code.

[daledupreez]

We rolled out the reduced 1 hour cache period to all WordPress.com Simple sites as of a few minutes ago in D85067-code. Podcasts may remain cached for up to 12 hours from now if they were already cached, but that should resolve by later today.

I will get started on the work to automatically set this up for WordPress.com Atomic sites (i.e. sites with plugins) so we can give them the same improvements next week.

[renata-franco]

Hi @daledupreez. Thank you for updating this! Does this mean the issue should be solved on all Simple sites?

[daledupreez]

Hi @renata-franco! Yes, we should have a 1 hour cache period for podcasts on all Simple sites.

We should be rolling out a similar fix to Atomic sites this week in Automattic/wpcomsh#1056.

[renata-franco]

Hi @renata-franco! Yes, we should have a 1 hour cache period for podcasts on all Simple sites.

That is such great news, Dale. I will update the user I am working with regarding this issue then. Thank you so much for working on this.

[daledupreez]

Over the course of the last day, we have rolled out the two sets of changes needed for our Atomic platform to reduce the cache timeout to 1 hour for the Podcast Player and Anchor.fm blocks. This means these blocks should only keep cached podcast information for an hour before checking for updates to the podcast RSS feed. I am going to mark this issue as fixed! 🎉

For reference, the two relevant changes were Automattic/jetpack#24966 (which was rolled out yesterday as part of a Jetpack 11.3 alpha release), and Automattic/wpcomsh#1056, which rolled out over the last few hours.

[synora]

While this issue is closed, I had to manually clear the cache for the user in 35765219-hc this morning. Follow-up is at 5381587-zd-woothemes

[Robertght]

I'm reopening this for now as it seems like the user is still facing this issue and they've grown impatient and frustrated.

@daledupreez can we get a double check on this specific case (5508841-zen) please?

[daledupreez]

Thanks for the ping and keeping things moving for the customer, @Robertght! 🙇

I've reviewed the situation and it looks like we didn't provide them with a very clear description of what we're doing and what impact our changes will have in the ticket they had open while we were working on these fixes, 5286257-zd-woothemes.

I think our best next step will be to reach out to the user to communicate a few things:

1. Communicate that we really want to help them and make sure podcasts work well for them
2. Share how podcasts currently work: to keep the site as fast as possible, we keep/remember the podcast episodes for 1 hour so we don't have to re-fetch it and keep users waiting. However, this means that there are times where it will take up to an hour for updates to flow through to the website. The exact timing depends on when we last fetched the podcast episode, but the rollout should always occur within an hour. (It's also worth noting that before our recent changes, we thought the update time was 1 hour, but it was actually 12 hours, which was way too long. 😬 )
3. Can we ask what timeframe they would like for updates to appear? The tradeoff here is that we want to keep the site nice and quick, and we picked 1 hour as a timeframe that would allow for somewhat timely updates without introducing delays for most site visitors. But we'd love to know what sort of timeframe would be acceptable to the user.
4. It may also be worth reframing the question a bit: how long is it taking for the podcast to appear on other systems? I'd like to make sure we understand the user's expectations, especially as we're not matching the behaviour they want/expect.

[KirkwallDay]

3. Can we ask what timeframe they would like for updates to appear? The tradeoff here is that we want to keep the site nice and quick, and we picked 1 hour as a timeframe that would allow for somewhat timely updates without introducing delays for most site visitors. But we'd love to know what sort of timeframe would be acceptable to the user.

4. It may also be worth reframing the question a bit: how long is it taking for the podcast to appear on other systems? I'd like to make sure we understand the user's expectations, especially as we're not matching the behaviour they want/expect.

Hey Dale, I checked with the user here and they would like it to be as low as 15 minutes if that is possible? I did explain to them the differences between this block and a podcast listing service, they say it propagates to those within a few minutes. I suspect the other podcast sites they reference are using different embeds and not using the podcast block, but I've asked for clarification on some of the other sites they mentioned so we can check that.

Is it possible to get this down to 15 minutes without affecting site performance too much, or is an hour the best we can do?

[daledupreez]

Thanks for the update, @KirkwallDay!

I will need to look into whether we can explore a different approach for the caching that allows us to check for updates more regularly. I don't think it makes sense to reduce the normal caching by too much, because we want the site to be fast, but we need a quicker way to check whether there has been more content.

I have an initial idea that I will explore next week. I will note any progress I make in this issue.

[renata-franco]

Thanks @daledupreez, @KirkwallDay, and @Robertght for responding to the user while I was AFK!

[KirkwallDay]

@daledupreez @renata-franco The user did get back to us and noted another one of their sites updates within minutes. I checked that site (URL is in the ticket 5508841-zen ) and it's a WordPress.com Premium site using our Podcast block as well. It's possible that if they upload those podcasts at the same time their site is just more on the tail end of the 1 hour caching window?

[daledupreez]

@KirkwallDay, I think you're definitely on the right track. If it helps explain the behaviour the user, every time we get a visitor to the page with the podcast block, we start by trying to reuse the podcast data from the last time we fetched the podcast, as long as it was within one hour. If you haven't had any visitors to the page within the last hour, we'll update the podcast immediately. But if someone visited the site 10 minutes ago, and you updated 5 minutes later, the update will likely take another 50 minutes to flow through.

I am continuing to explore a shorter timeframe, but will only get to testing that a bit more tomorrow.

[renata-franco]

I am continuing to explore a shorter timeframe, but will only get to testing that a bit more tomorrow.

Hi, hi @daledupreez. I am checking on this new ticket the user created. In the past, I was manually clearing the cache every Friday at 12 CEST to avoid any further issues. However, I wanted to check in with you to know if the possibility of incrementing the cache thingy to 15 minutes is actually possible. If you say it is, I can continue clearing the cache for the user in the meantime. Let me know, please.

[daledupreez]

@renata-franco, I've been exploring this and pushed up an initial PR in https://github.com/Automattic/jetpack/pull/26097. This still needs feedback from the Jetpack teams, so we may or may not get these changes shipped, but I will note any progress or decisions in this issue. More specifically, I think this is something I think we should handle better, but I think my approach may need to be refined.

[renata-franco]

Thanks for the info @daledupreez. Do you think I should check on the site and clear cache manually if needed in the meantime or should I not offer that in this case?

[daledupreez]

Hi @renata-franco, I don't think my change is going to land before tomorrow, so we are unlikely to have an improvement in place for the next update tomorrow.