istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.17k stars 323 forks source link

Fix online dmoz test #257

Closed madisonb closed 3 years ago

madisonb commented 3 years ago

dmoz should really be replaced for the entire crawling documentation/setup. Hopefully this outage is only temporary, but this PR fixes the build scripts so the containers build once again.