search

diegov / searchbox

Personal crawling and indexing
GNU General Public License v3.0
2 stars 0 forks source link

Add option to process cached results. #24

Closed diegov closed 2 years ago

diegov commented 2 years ago

Add option to process cached results.

Some spiders will crawl paginated results starting on page 1, and following next links, so if we skip processing the first page because it's unmodified, we might not see the last page which contains new entries.