search

Niger-Volta-LTI / iranlowo

Ìrànlọ́wọ́ is a utility library for analysis & (pre)processing of Yorùbá text → https://pypi.org/project/iranlowo
MIT License
17 stars 8 forks source link

[ADD] an module for scapers #1

Open ruohoruotsi opened 5 years ago

ruohoruotsi commented 5 years ago

Add a module that does Yorùbá TS website scraping. Right now candidates are:

Olamyy commented 5 years ago

While thinking about the best way to implement this, I came up with a design and would like to get opinions about it.

Since we want the crawler to be reusable, I'm proposing a minimal https://docs.scrapy.org app that contains:

  1. A spider : BaseCrawler that inherits from scrappy.Spider.
  2. A settings for handling scrapy configs
  3. An item item file for handling ordered saves.

The BaseCrawler from 1 above can implement a yaml/json loader for any other person to subclass to run create their own config/logic.

I imagine this is a bit confusing right now but I can come up with a simple demo of my thoughts and share it for opinions here when I'm done.

realsama commented 5 years ago

@ruohoruotsi so i see there are plans for a scraper already, what is the progress so far cause am thinking of one that scrapes bbc yoruba although it can be sub classed and tuned for other blogs and websites generally.