search

sc3 / cookcountyjail

A Django app that tracks the population of Cook County Jail over time and summarizes trends.
http://cookcountyjail.recoveredfactory.net/api/1.0/?format=json
Other
31 stars 23 forks source link

Decouple scraper from other administrative tasks #336

Open bepetersn opened 10 years ago

bepetersn commented 10 years ago

This is a list of all the types of tasks that need to be done on the server, currently totaling to 19 different kinds of things; I got this list from compiling all the things we have talked about in the issues, looking at everything the scraper.sh file does currently, plus using my imagination:

https://docs.google.com/document/d/1ep3Kv_F_7xpePbAOwBusMj0aJ_J3qn8qLAeFJA17qCU/edit

All of these things need to be done, not by the scraper, but by a program which is always running on the server, and which itself can start the scraper and other tasks concurrently (or in a blocking fashion, depending).

bepetersn commented 10 years ago

This is part of the attempt to do #67.