eklem / browsercrawler

Crawling content from a site within the browser. A basis for i.e. a search solution for static sites.
https://eklem.github.io/browsercrawler/doc/
MIT License
2 stars 0 forks source link

Do all crawls (fetch) and cheerio-stuff before adding to search index #17

Open eklem opened 6 years ago

eklem commented 6 years ago

https://daveceddia.com/waiting-for-promises-in-a-loop/

Use the second example, then it's easier to add a play-nice timer between each fetch() request:

var chain = $q.when();
for(var i = 0; i < 5; i++) {
    chain = chain.then(function() {
        return $http.get('/data' + i);
    });
}

After the foor-loop a search-index adder can be called.

eklem commented 6 years ago

Actually, no. Do one and one, and let search-index-housekeeper keep track of whats indexed and not. We don't know when the user will click to a new page.

eklem commented 6 years ago

Or if an array: https://pouchdb.com/2015/05/18/we-have-a-problem-with-promises.html

eklem commented 6 years ago

Check promise.all