search

OP-Engineering / link-preview-js

⛓ Extract web links information: title, description, images, videos, etc. [via OpenGraph], runs on mobiles and node.
MIT License
770 stars 124 forks source link

Attention Required! | Cloudflare #136

Closed wouterlemcke closed 2 years ago

wouterlemcke commented 2 years ago

Some url's successfully return a link preview object. However the title is Attention Required! | Cloudflare, which we don't see when we open the page in the browser.

As far as I can see there are no redirect or anything on the page I'm trying to load. Is there something I can do to fix this?

An example url is: https://bible.com/bible/1990/rom.8.31.HSV

ospfranco commented 2 years ago

Cloudfare bot detection has probably marked your IP address or somehow detected a bot and is blocking your requests. You can try to play with the user agent as some of the big corpos are whitelisted (Apple uses facebook's and twitter's). Nothing wrong with the library, I would suggest asking on Stack Overflow, some ppl might have some ideas.

ospfranco commented 2 years ago

Ah sorry, I misread you do get a response, but with the wrong title. No idea, each page can return w/e they like, this just reads the tags.

sergei-zelinsky commented 1 year ago

For anyone experiencing this issue in the future. I did overcome it using the Slackbot user agent header.

For example:

const preview = await getLinkPreview(url, {
  followRedirects: 'follow',
  headers: {
    'user-agent':
      'Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)',
  },
});