RSS Feed stopped working

[JokerGermany]

Since a couple of days I am wondering about missing RSS Messages.

https://matrix.to/#/#messenger-de:matrix.org https://matrix.org/blog/feed 9.6.2023, 06:45:17:✅ Successful fetch Which Timezone? Really nearly 20 hours ago? I am missing the last TWIM.

Other example: https://matrix.to/#/#homeassistant-de:matrix.org https://matrix.to/#/#Home-Assistant:matrix.org https://github.com/home-assistant/core/releases.atom There have been no recent updates for this feed. The Feed was Added 2023-06-04 Since the adding that time there were 4 Releases, no Release showed up in both rooms.

[drskalman]

same here. my last sucessful fetch is 11.6.2023 is two days ago basically.

[Twi1ightSparkle]

May possibly be resolved by https://github.com/matrix-org/matrix-hookshot/pull/779

[Danie10]

Still not working for me. I have tried recreating my feeds, and also ticking the box to report feed failures (nothing gets reported).

[JokerGermany]

Could you pls switch back to the old RSS Feed, which runs smooth without having to be worried that you missed a new Entry or without more than a week long outage.

Just switch back to the old RSS Feed and switch to the new one when you can use it... Perhaps let both system run at the same time and let people test the new one to see if it's able to be productive and find bugs to make it production ready...

Is that the new Element strategy? Releasing broken software to replace stable software? :scream:

[Danie10]

@JokerGermany do you mean the depreciated one? It had stopped working at one point, and I had removed it from room. How do I invite it back now as current instructions seem to be for new one. The new one did start working, then stopped, then started again and stopped. So yes, if the old one would still work I'd like to use it as my channel is dead right now.

EDIT: It shows now as Invited to the room, but I'm trying to figure out how to get a response from it.

[Twi1ightSparkle]

We have deployed a new Hookshot version which includes https://github.com/matrix-org/matrix-hookshot/pull/779 My feeds have started working again. Is it working for you all?

The old feeds bot is deprecated and turned off. You won't get any response from it.

[JokerGermany]

If you mean it should be fixed after the message from last week i can definitly say NO. I chatted with other users which rss feed isn't working. For https://matrix.org/blog/feed I got one feed Message (A Brand new website), nothing more. The TWIM from last Friday didn't appeared. But it said that it made a successful fetch today: 20.6.2023, 14:43:08:✅ Successful fetch

For the other Example: https://github.com/home-assistant/core/releases.atom 20.6.2023, 14:43:21:✅ Successful fetch But I never got a message Last Release Message was 5 Days ago.

Last Friday i subbed to https://github.com/bramstroker/homeassistant-powercalc/releases in https://matrix.to/#/#ha-releases:matrix.org 6 hours ago a new release came out, but nothing is reported until now. The Feed tells me:

There have been no recent updates for this feed.

[mfilipe]

We have deployed a new Hookshot version which includes matrix-org/matrix-hookshot#779 My feeds have started working again. Is it working for you all?

The old feeds bot is deprecated and turned off. You won't get any response from it.

It isn't working in my case. Actually, it is unstable because sometimes it returns posts but misses a lot of them.

[Danie10]

One post just went through me now. Maybe it has started working again.

[Twi1ightSparkle]

Thanks for the feedback. Can confirm that I also see some posts go missing occasionally. We're continuing investigating the root cause of this

[JokerGermany]

Today in https://matrix.to/#/#matrixgeeks:matrix.org the feed pot posted a Release Tag from 3 days ago^^ https://github.com/vector-im/element-desktop/releases/tag/v1.11.34

[bkil]

It has been more than 2 weeks now. After the issue had been resolved, could you perhaps publish a post mortem so we can learn from it?

I agree that in general, you should not switch off good working alternatives until the replacement is stable and has all features implemented. Anticipating such bugs and seeing the constant stream of new bug reports, I tried to hold out as long as possible on the old bot, but had to forcefully migrate after you switched the good one off.

[Half-Shot]

Hi there :wave:

We've been working on this issue for quite some time and I do appreciate the points made about the migration being flaky. The issues we encountered here were scaling ones and I'm glad to say that after working on https://github.com/matrix-org/matrix-hookshot/pull/786, we seem to be at a stable point (with lots of room to grow).

A change made a few weeks ago caused Hookshot to dramatically increase in memory which caused the bot to have regular outages, which in turn caused the feeds to become unreliable. At present, we handle something like ~2500 feeds in the bot. It's not a lot by any means, but it's also a lot more than Hookshot had been rated for under internal and commmunity deployments.

After having everyone migrate their feeds from Go-Neb to Hookshot, we found that the library used by Hookshot was incredibly unforgiving and inflexible to certain feeds that the Go-Neb bot was happy to process. This was one of the things we quickly learnt about RSS/Atom feeds -- they're surprisingly easy to screw up. We had done a fair amount of testing against a sample of RSS feeds, but certainly that is nothing compared to the variety provided by the Matrix community.

Since we're already working on efforts to port Hookshot to Rust, and Hookshot needs to scale: We ported the feed parsing logic to Rust. This massively improved our feed reliability...for a time. We found that the number of feeds we were now processing was increasing past the point where a single "thread" could keep up, which was quickly solved by adding some concurrency. The concurrency however gave way to a small but compounded memory leak between the Rust and Node layers of Hookshot.

Which is where we got to about 3 weeks ago, where the bot started to fail to keep up with feeds and persistently restarted. As I say above, we believe we've now gotten to the bottom of it and we'll continue to watch for more issues as they come.

[Half-Shot]

Worth saying that if you are still experiencing issues, feel free to drop a line here and I'll investigate.

[Half-Shot]

I'm going to close this as fixed with the release of https://github.com/matrix-org/matrix-hookshot/releases/tag/4.4.0

[JokerGermany]

It's happening again: https://matrix.to/#/#messenger-de:matrix.org https://matrix.org/blog/2023/07/postponing-libera-chat-deportalling/#continue-reading @Half-Shot

[Danie10]

Yes, mine posted last yesterday, but not for today's post 7 hours back.

[Twi1ightSparkle]

That is odd, I have a testing/monitoring feed that posts every 10 minutes. It's only missed three in the past month. It posted a post 9 minutes ago

[bkil]

I confirm to be encountering missing RSS updates. You can see for yourself by comparing the RSS entries posted in the matrix rooms vs. the RSS XML changes committed to the git repo being followed

• https://github.com/bkil-bot/osm-rss-juggler/commits/gh-pages
• https://matrix.to/#/#osmhungarylive:grin.hu

The following xml files are hooked up to this single room (about 90% of the updates): changeset-discuss, gitlab-piclayer, mastodon-doudouosm, mastodon-openstreetmap, osm-notes, wiki.