search

michaelhelmick / lassie

Web Content Retrieval for Humans™
https://lassie.readthedocs.org
MIT License
614 stars 49 forks source link

Fake user agent #56

Closed michaelhelmick closed 7 years ago

michaelhelmick commented 7 years ago

https://www.amazon.com/gp/aw/d/1499803125/ref=ya_aw_od_pi?ie=UTF8&psc=1 returns a "Robot Check" title...

By default, we should fake the user agent, so websites don't think lassie is a robot (she's a dog).

https://github.com/hellysmile/fake-useragent

michaelhelmick commented 7 years ago

I think I will opt to hardcode the user agent by default. Others can change the user agent if need-be via setting header in Lassie.request_opts