search

the-radiativity-company / inmates

a tool for collating inmate rosters
2 stars 1 forks source link

Set up testing framework for spiders #9

Closed theradiativitycompany closed 4 years ago

theradiativitycompany commented 4 years ago

Currently spiders run against a live site. It would be convenient if they could run against a locally-cached version of the site. This will reduce developer feedback time as well as help the project keep a low profile.

theradiativitycompany commented 4 years ago

some looks on testing from city-scrapers:

easherma commented 4 years ago

We also did learn that the 'url' each spider takes can also be a file URL.

withtwoemms commented 4 years ago

closing since merged