search

Pinjasaur / meed

Get Medium RSS feeds as JSON.
https://meed.js.org/
5 stars 2 forks source link

403 Forbidden #1

Open ImTheDeveloper opened 2 years ago

ImTheDeveloper commented 2 years ago

Wondering if something has changed, when attempting to run a feed now I'm getting a 403 Forbidden using node.js. When hitting the medium feed url in my browser it loads fine.

Anyone else noticed this?

Pinjasaur commented 2 years ago

Hey Chris, thanks for taking the time to report this bug.

I spent some time digging into it and was able to reproduce the issue you describe. Unfortunately, it appears that for whatever reason node-fetch is triggering the Cloudflare Captcha on those medium.com requests. The workaround I was able to find was specifying an HTTP User-Agent header other than the default node-fetch e.g.

const fetch = require("node-fetch")

;(async () => {
  fetch('https://medium.com/feed/@Medium', {
    headers: {
      'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0'
    }
  })
    .then(res => res.text())
    .then(res => console.log(res))
})()

I've began work in the v2 branch (https://github.com/Pinjasaur/meed/tree/v2) to generate a user agent string custom to meed that includes the version number.

Unfortunately, while digging into this bug report I also noticed that medium.com/topics is now totally deprecated and they have merged the notion of topics and tags into singular entity now referred to as tags. Because of this, the entire Meed#topics() API is now obsolete which is why I'm placing this work on the aforementioned v2 branch (because it's a breaking change).

I don't have an immediate workaround I can suggest to you until I get a new version published to npm. Sorry about that. Hoping I can get a new version published relatively quick.