In use with a search index: When to do a recrawl?

eklem / browsercrawler

Crawling content from a site within the browser. A basis for i.e. a search solution for static sites.

https://eklem.github.io/browsercrawler/doc/

MIT License

2 stars 0 forks source link

In use with a search index: When to do a recrawl? #32

Closed eklem closed 4 years ago

eklem commented 6 years ago

You need two dates:

When was content last updated (updated on all pages)
When was content last crawled (needs to be stored in browser)

eklem commented 6 years ago

Think I need to write the ID (url) of every document indexed. This when there is a lot of documents, or using the generic crawler. It will take some time, and maybe more time than the user spends on the first page.

eklem commented 6 years ago

search-index-housekeeper will handle this