Is there a way to only crawl new pages?

VIDA-NYU / ache

ACHE is a web crawler for domain-specific search.

http://ache.readthedocs.io

Apache License 2.0

444 stars 135 forks source link

Closed 0xEnders closed 7 months ago

0xEnders commented 7 months ago

Hi there, love the crawler so far!

I would like to check if there's a way to compare the results of my new crawl and the old one and only save the results of the new crawl?

Thanks!