search

ruipgil / scraperjs

A complete and versatile web scraper.
MIT License
3.71k stars 188 forks source link

Return full request response rather than just the Cheerio object #14

Closed ashooner closed 10 years ago

ashooner commented 10 years ago

I'd like to suggest passing the full response to the processing function rather than just the cheerio document object, since some scraping will want access to headers or other information elsewhere in the response.

ruipgil commented 10 years ago

Cheerio is only for scraping the DOM, you can access the those elements by using the utils object.

Scraper.create(url)
  .then(function(utils) {
    console.log(utils.scraper.response);
  });

and that will log the response object returned by request.