search

eklem / browsercrawler

Crawling content from a site within the browser. A basis for i.e. a search solution for static sites.
https://eklem.github.io/browsercrawler/doc/
MIT License
2 stars 0 forks source link

Follow links: true/false #50

Open eklem opened 6 years ago

eklem commented 6 years ago

Make it possible to not scrape content for links, and then not add to URL queue. Together with the possibility to add several URLs, this will make possible the use case where you know all the URLs you want the users to crawl up-front.