Open ruohoruotsi opened 5 years ago
While thinking about the best way to implement this, I came up with a design and would like to get opinions about it.
Since we want the crawler to be reusable, I'm proposing a minimal https://docs.scrapy.org app that contains:
The BaseCrawler from 1 above can implement a yaml/json loader for any other person to subclass to run create their own config/logic.
I imagine this is a bit confusing right now but I can come up with a simple demo of my thoughts and share it for opinions here when I'm done.
@ruohoruotsi so i see there are plans for a scraper already, what is the progress so far cause am thinking of one that scrapes bbc yoruba although it can be sub classed and tuned for other blogs and websites generally.
Add a module that does Yorùbá TS website scraping. Right now candidates are: