search

krthush / link-preview

An API for getting near perfect link preview data - works for Node.js enviroments.
https://favorited-link-preview.herokuapp.com/
MIT License
42 stars 19 forks source link

few problem sites - https://www.bloomberg.com/ https://reebok.com/ #3

Closed krthush closed 3 years ago

krthush commented 3 years ago

still there a few problem cases (for example when I tried https://www.bloomberg.com/, https://reebok.com/, I run into some problems - i imagine bot detection / redirect to sorry pages), how should i go about trying to bypass some of these sites? for my use case i'm only try to get a page title, description and image (much like we get link previews for signal, whatsapp, etc.), still some sites are really tricky to get this off of? any thoughts?

krthush commented 3 years ago

so this is interesting - would be cool if mention in the extra / stealth docs, it seems like using the AdblockerPlugin helps with bot detection?

see 8c75e5a

at least for the case of https://reebok.com/, I'm now able to fetch some data after using .use(AdblockerPlugin({ blockTrackers: true })), bloomberg still a problem case