bdevos / rss-to-email

Scheduled RSS to Email with Github Actions
MIT License
31 stars 26 forks source link

"RSS Feeds Returning Error - Unable to Fetch Feeds from LowEndTalk" #347

Open thelittlerocket opened 2 weeks ago

thelittlerocket commented 2 weeks ago

Hi there, thanks a bunch for this awesome script! I’m trying to use it with two RSS feeds: https://www.lowendtalk.com/categories/domains/feed.rss and https://www.lowendtalk.com/categories/offers/feed.rss. But it seems like the script isn’t getting the feeds correctly. It works fine when I access them directly in my browser, but I can’t figure out where I went wrong. Do I need to make any changes to the configuration or is there something wrong with the feeds themselves? Let me know!

Here’s how I’ve set up my feeds in the configuration:

export const feeds = [
  // LET Offers
  'https://www.lowendtalk.com/categories/offers.rss',

  // LET Domains
  'https://www.lowendtalk.com/categories/domains.rss',

]

And here’s the action result:

Run npm run email actionUrl=https://github.com/thelittlerocket/rss-to-email/actions/runs/11676138054 lastSuccess=
  npm run email actionUrl=https://github.com/thelittlerocket/rss-to-email/actions/runs/11676138054 lastSuccess=
  shell: /usr/bin/bash -e {0}

> email
> node email actionUrl=https://github.com/thelittlerocket/rss-to-email/actions/runs/11676138054 lastSuccess=

Could not settle feed https://www.lowendtalk.com/categories/offers.rss, reason: Error: Status code 403
Could not settle feed https://www.lowendtalk.com/categories/domains.rss, reason: Error: Status code 403
Feed https://www.lowendtalk.com/categories/offers.rss failed, reason: Error: Status code 403
Feed https://www.lowendtalk.com/categories/domains.rss failed, reason: Error: Status code 403
Error: One or more feeds failed while no new items!
    at Module.filterItemsFromFeed (/home/runner/work/rss-to-email/rss-to-email/src/filterItems.ts:23:9)
    at renderEmail (/home/runner/work/rss-to-email/rss-to-email/src/renderEmail.tsx:21:47)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async createEmail (file:///home/runner/work/rss-to-email/rss-to-email/email.js:19:33)
Error: Process completed with exit code 1.

I’m having a bit of trouble with my Node.js script. It seems to be stuck at the Module.filterItemsFromFeed function, and I can’t seem to figure out why. I’ve been trying to debug it for hours, but I’m not making any progress.

Here’s the error message I’m getting:

at Module.filterItemsFromFeed (/home/runner/work/rss-to-email/rss-to-email/src/filterItems.ts:23:9)
at renderEmail (/home/runner/work/rss-to-email/rss-to-email/src/renderEmail.tsx:21:47)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async createEmail (file:///home/runner/work/rss-to-email/rss-to-email/email.js:19:33)
Error: Process completed with exit code 1.

I’m hoping you can help me figure out what’s going on and get this script running again. I’ve been working on this for a while now, and I’m really frustrated that I’m stuck at this point.

Thanks so much for your help!

bdevos commented 1 week ago

The problem appears to be that those feeds seem to be blocked by some Cloudflare security.

In the output there is this line:

Could not settle feed https://www.lowendtalk.com/categories/offers.rss, reason: Error: Status code 403

When I do a curl to just the RSS feed I also get the same 403 which the RSS parser runs into:

curl 'https://lowendtalk.com/categories/offers.rss'

When I provide a User-Agent header with curl, it does return a proper RSS response:

curl 'https://lowendtalk.com/categories/offers.rss' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36'

I tried just adding that header into the RSS parser, but it did not work. At first I thought that maybe the header wasn't properly send. But when I request something else I see the request header coming through just fine.

There seems to be something in Cloudflare that detect these requests and blocks them with a 403.

I did a quick search if there are workarounds, but non seem very simple to just adjust in the request itself. When googling if other people run into this issue with those feeds I see a bunch of posts, but I have not yet found any suggestion that resolves the problem.