search

tusharojha / web_scraper

A very basic web scraper implementation to scrap html elements from a web page.
https://pub.dev/packages/web_scraper
Apache License 2.0
77 stars 42 forks source link

The output is different to what I see in the browser #69

Open theRealBitcoinClub opened 2 years ago

theRealBitcoinClub commented 2 years ago

When I visit a page in the browser with that following URL then I get a different output as to when I use the Scraper getPageContent() method

e.g. view-source:https://www.google.com/search?q=

delivers a different output than

webScraper.getPageContent().toString()

Is that because of some USER-AGENT settings and Google is preventing the scrape? How can I solve that issue?

tusharojha commented 2 years ago

Yes probably, I have to add an option to set custom USER-AGENT.