search

Datenschule / jedeschule-scraper

MIT License
22 stars 15 forks source link

Add Scrapy option to non-scrapy crawlers? #42

Closed cyroxx closed 4 years ago

cyroxx commented 4 years ago

For the sake of an experiment, I converted my WIP crawler for Hessen into a Scrapy spider: cyroxx/jedeschule-scraper@0263e41711eeddc7ca57e8c460f2bb898f59f77a

Is this an approach that we should consider for those crawlers that are not yet scrapy based - at least as a second option? This would allow us to implement scrapy pipelines with them and use them on ScrapingHub.

k-nut commented 4 years ago

I think for the sake of having all the scrapers run through the pipelines and on scraping hub this is a really good idea!

knutator2 commented 4 years ago

+1

knutator2 commented 4 years ago

can we close this one? @cyroxx @k-nut

k-nut commented 4 years ago

Yes, I think we have an agreement that this is a good idea ;)