Add User-Agent string to requests

gabceb / node-metainspector

Node npm for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, an array with all the links, all the images in it, etc. Inspired by the metainspector Ruby gem

MIT License

129 stars 52 forks source link

Add User-Agent string to requests #22

Closed alexymik closed 8 years ago

alexymik commented 8 years ago

Some sites (such as Facebook) return an invalid or otherwise unexpected page if no user agent is set.

gabceb commented 8 years ago

Covered via #27 since all options provided are now passed to request