search

amoilanen / js-crawler

Web crawler for Node.JS
MIT License
253 stars 55 forks source link

Evaluate selectors #34

Open BrunoQuaresma opened 7 years ago

amoilanen commented 7 years ago

This may be an interesting feature, however we will need to parse the pages and maybe should consider using some additional HTTP client in addition to the one provided by 'request'.

Currently crawler is not quite a browser, neither it uses one to load the pages it crawls. But this can be interesting to research further. For example, by default the crawler can use the present lightweight non-browser mode but when needed switch to full-fledged browsing. I imagine this can be useful in, for example, discovering JavaScript errors on the crawled pages during testing.

BrunoQuaresma commented 7 years ago

Thanks for the answer.

amoilanen commented 7 years ago

Thanks for a good idea, would be nice to try it out some time in the future.