Closed pjsier closed 4 years ago
A lot of the time used in sending scraped results to the Wayback Machine is spent on links that are several years old. This adds a CITY_SCRAPERS_ARCHIVE settings that only scrapes these meetings if it's flagged and applies it to a few spiders
CITY_SCRAPERS_ARCHIVE
A lot of the time used in sending scraped results to the Wayback Machine is spent on links that are several years old. This adds a
CITY_SCRAPERS_ARCHIVEsettings that only scrapes these meetings if it's flagged and applies it to a few spiders