search

pipes-digital / pipes

Repository for Pipes
https://pipes.digital
GNU Affero General Public License v3.0
254 stars 21 forks source link

Feed could not be parsed #59

Closed anewuser closed 4 years ago

anewuser commented 4 years ago

There's no output for this feed on Pipes:

https://www.pipes.digital/pipe/LOMGJx9r

https://old.reddit.com/search.rss?q=linux&restrict_sr=&sort=top&t=day

https://www.simplepie.org/demo/?feed=https%3A%2F%2Fold.reddit.com%2Fsearch.rss%3Fq%3Dlinux%26restrict_sr%3D%26sort%3Dtop%26t%3Dday

This works, though: https://old.reddit.com/search.rss?q=linux

onli commented 4 years ago

I had a look at this. For me it works, sometimes, locally. But already there I get too many 429 Too many Requests errors to properly debug this. I think that reddit put a very strict rate limiting on those sorted search RSS feeds, maybe even not per feed put globally. And there seems to be some user agent sniffing going on, because I don't get any thus notices in the browser.

If it gets blocked by the rate limiting that empty output will be cached for a while, if multiple pipes battle for a global rate limiting I can't see this end well.

I don't see a good solution here. There is probably a way to properly manage the quota reddit gives, but nothing I can act on yet.

anewuser commented 4 years ago

I see. I thought it could be some parsing error. Since this a limit imposed by Reddit, and I can just create a bookmark instead, I'm going to close this. Thank you for looking into it!

onli commented 3 years ago

@anewuser After several other approaches I now added a reddit block that uses rss-bridge to get a properly cached reddit feed. This worked stable for me so far. If you are still interested you could try those feeds again :)

anewuser commented 3 years ago

@onli Is it possible to set the block to use custom sorting parameters? I want to get only the top posts from the past week.

Example with a built-in feed: https://old.reddit.com/r/linux+firefox/top/.rss?sort=top&t=week

onli commented 3 years ago

The reddit block is using rss-bridge, and I don't see a way to set a a sort parameter there. Neither in the UI nor the code. I might miss something of course.

I reworked how Pipes downloads sites and feeds, in https://github.com/pipes-digital/pipes/commit/289a9e77cc14c9662fd7f9dbbca334a373e3798a - could be worth trying again whether Reddit feeds work directly now, with just a feed block.

anewuser commented 3 years ago

They're still rate-limiting Pipes. This makes it harder to test filters, but I think the output will still work, since my reader won't check for updates that often.

onli commented 3 years ago

I made a follow-up change, trying a custom user agent. With the reddit API that's a big thing, might help here as well.