Crawler main executor development

jijames / electionWatch

A collection of crawlers and social media analysis tools to track and investigation concept trends

2 stars 0 forks source link

Closed jijames closed 7 years ago

jijames commented 7 years ago

Either shell / python / Go develop a main crawler controller app

Each crawler will be a separate module that the main app auto detects and executes
Controller should be able control the timing/speed of each crawler
Controller should manage the output and settings locations of each crawler

Start with an initial crawler on joonang daily as an example. Current crawler script can be a starting point.

jijames commented 7 years ago

@lukenukem95 No updates last week? What is your plan for this?