search

niklasb / dryscrape

[not actively maintained] A lightweight Python library that uses Webkit to enable easy scraping of dynamic, Javascript-heavy web pages
http://dryscrape.readthedocs.io/
MIT License
533 stars 67 forks source link

user-agent switching #14

Closed osleg closed 10 years ago

osleg commented 11 years ago

Hey, Just a short question, how do I change user-agent in dryscraper?

Sorry if this is wrong place to ask but didnt find other places :)

niklasb commented 11 years ago

I don't think this is implemented. You can try what happens if you use webkit_server.Client.set_header to change the User-Agent header, but I'm not sure if this will work as expected.

osleg commented 11 years ago

I tried it already but it sends 2 headers, one with my User-Agent and second original

niklasb commented 11 years ago

You'll have to hack webkit_server to achieve what you want then. I'll remember to include the option to overwrite header values, but I don't know when I'll find the time.

On Wed, Dec 5, 2012 at 12:49 PM, Akira Kir notifications@github.com wrote:

I tried it already but it sends 2 headers, one with my User-Agent and second original

— Reply to this email directly or view it on GitHubhttps://github.com/niklasb/dryscrape/issues/14#issuecomment-11038516.

niklasb commented 10 years ago

set_header now has a switch to set the user agent properly.