Now whenever a changed page is detected, we crawl all the local URLs
linked to by that page. This should help pick up site-wide changes
and changes in pagination page contents.
We skip adding local URLs to the queue if the page content has not
changed since it was last cached. This change should speed up crawling
even without the option enabled.
Now whenever a changed page is detected, we crawl all the local URLs linked to by that page. This should help pick up site-wide changes and changes in pagination page contents.
We skip adding local URLs to the queue if the page content has not changed since it was last cached. This change should speed up crawling even without the option enabled.