search

eklem / browsercrawler

Crawling content from a site within the browser. A basis for i.e. a search solution for static sites.
https://eklem.github.io/browsercrawler/doc/
MIT License
2 stars 0 forks source link

Add URL(s) to URL-queue before crawling start #41

Closed eklem closed 5 years ago

eklem commented 6 years ago

Make the flow so that when you feed a URL (or several URLs) to be crawled, it is first added to the pouchDB crawl-Queue. This way, an array of URLs or one URL will be easy to accept with the same functionality. Should be able to have a "noFollow"-flag so that you can feed an array of all URLs to be crawled.

eklem commented 5 years ago

Should only have one URL as starting point.