Some websites don't get scraped if i pass useragent request headers

website-scraper / website-scraper-puppeteer

Plugin for website-scraper which returns html for dynamic websites using puppeteer

MIT License

324 stars 80 forks source link

Some websites don't get scraped if i pass useragent request headers #43

Closed shreejalmaharjan-27 closed 1 year ago

shreejalmaharjan-27 commented 2 years ago

Some websites, eg: google.com don't get scraped if i pass custom useragent headers request: { headers: { 'User-Agent': "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_15) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4859.164 Safari/537.36" } },

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.