ruipgil / scraperjs

A complete and versatile web scraper.
MIT License
3.71k stars 188 forks source link

User-id #56

Closed QuentinRoy closed 8 years ago

QuentinRoy commented 8 years ago

It is sometimes useful to be able to set the user-id/user-agent for the request as the target website may depend on it.

ruipgil commented 8 years ago

We use requestbehind the scenes, which allows you to do it very easily.

Check this post out. More specifically, the section Be a great lad.

QuentinRoy commented 8 years ago

Oh I see. Thank you @ruipgil.

As a suggestion, it could be worse documenting it, especially if it is a supported behavior. Even in the code, the get argument is only named "url". One has to guess that as it is directly forwarded to request, it may be replaced with something else, and look at request documentation to see what is actually possible.

Also, it is possible that there is an error in this post. request’s headers option finishes with an “s”.