Addition of Googlebot profile to circumnavigate internet paywalls

dillbyrne / random-agent-spoofer

Firefox addon - Rotates complete browser profiles ( from real browsers / devices ) at a user defined time interval. It includes many extra privacy enhancing options

https://addons.mozilla.org/en-US/firefox/addon/random-agent-spoofer/

GNU General Public License v3.0

615 stars 108 forks source link

Addition of Googlebot profile to circumnavigate internet paywalls #369

Open sueridgepipe opened 8 years ago

sueridgepipe commented 8 years ago

Details on accessing web content behind paywalls...

http://www.ghacks.net/2016/02/26/read-articles-behind-paywalls-by-masquerading-as-googlebot/

It references two addons, RefControl and User Agent Switcher, but I think this would be an easy addition into RAS with a selectable googlebot profile.

Thoughts?

RoxKilly commented 8 years ago

Interesting. The capabilities discussed (spoofing referer and user agent) are close to what RAS already does, but I don't think developing tools for the purpose of circumventing paywalls fits the RAS mission. RAS' spoofing powers are meant to defeat tracking; this seems out of scope to me. @dillbyrne what do you think

berrythesoftwarecodeprogrammar commented 8 years ago

i think all they are asking for is for the useragent profile of googlebot to be added so one can use it when required

forkoz commented 8 years ago

Yes... we should have other crawler and bot profiles and how about adding custom profiles too.

agentxray commented 8 years ago

Definitely! This is in scope. Any common profile would add to our anonymity, and bots are quite common. I always enjoyed the bot useragents in Secret Agent. They could possibly be used for circumventing things other than paywalls too. Custom profiles would also be good to have.