search

matthewmueller / x-ray

The next web scraper. See through the <html> noise.
MIT License
5.88k stars 348 forks source link

useragent support? #188

Open bonesoul opened 8 years ago

bonesoul commented 8 years ago

is there a way we can set our own user-agent during requests? couldn't find an example

kdepp commented 8 years ago

I think you can write your own driver, in which you can set the useragent. Taking a look at the default http driver should help. https://github.com/lapwinglabs/x-ray-crawler/blob/master/lib/http-driver.js

johannish commented 8 years ago

Note that this is a duplicate of #168