search

RLesur / crrri

A Chrome Remote Interface written in R
https://rlesur.github.io/crrri/
Other
157 stars 12 forks source link

websites that don't like to be scraped #112

Closed staropram closed 1 year ago

staropram commented 1 year ago

Any tips on how to deal with websites that don't like to be scraped and have a service in front of them keeping an eye out for automated agents? What distinguishes a headless browser from a normal browser from the perspective of the server?

staropram commented 1 year ago

changing the user agent seemed to do the trick :)