sc3 / cookcountyjail

A Django app that tracks the population of Cook County Jail over time and summarizes trends.
http://cookcountyjail.recoveredfactory.net/api/1.0/?format=json
Other
31 stars 23 forks source link

Scraper failed to make a raw data dump on some occasions #452

Open bepetersn opened 10 years ago

bepetersn commented 10 years ago

This occurred on June 27th. Somehow, the completion (or creation) of the data dump did not happen, despite the fact that the scraper itself seemed to have completed its run. However, I judged this in a cursory albeit empirical manner based on there being no unusual errors that stood out in the log file for the day, and it may not be true. I will look into this some more.

In the near future, when #448 is done, we can recreate all the data that does exist in the v1 database (201 inmate records, ... an amount also not unusual).

nwinklareth commented 10 years ago

The June 11th scrape was also unsuccessful. The temp files still exist in the website/scratch/scraper/raw_inmate_data directory. They look to be smaller than then the normal ones. If you examine the log files for these days, which by the way are June 12th and 28th respectively you will find that there are index out of range errors exceptions thrown from InmateDetails.

On Mon, Jun 30, 2014 at 6:55 PM, Brian Everett Peterson < notifications@github.com> wrote:

This occurred on June 27th. Somehow, the completion (or creation) of the data dump did not happen, despite the fact that the scraper itself seemed to have completed its run. However, this was empirically judged based on there being no unusual errors noted in the log file, and may not be true. I will look into this some more.

In the near future, when #444 https://github.com/sc3/cookcountyjail/issues/444 is done, we can recreate all the data that does exist in the v1 database (201 inmate records, ... not an unusual amount).

— Reply to this email directly or view it on GitHub https://github.com/sc3/cookcountyjail/issues/452.

Regards

Norbert

Norbert Winklareth