Closed theradiativitycompany closed 4 years ago
Currently spiders run against a live site. It would be convenient if they could run against a locally-cached version of the site. This will reduce developer feedback time as well as help the project keep a low profile.
some looks on testing from city-scrapers:
We also did learn that the 'url' each spider takes can also be a file URL.
closing since merged
Currently spiders run against a live site. It would be convenient if they could run against a locally-cached version of the site. This will reduce developer feedback time as well as help the project keep a low profile.