Store log of successful scraper job

Datafable / epu-index

EPU index

http://www.applieddatamining.com/cms/?q=content/economic-policy-uncertainty-index

1 stars 0 forks source link

Store log of successful scraper job #64

Closed bartaelterman closed 9 years ago

bartaelterman commented 9 years ago

The web application needs to know how many scrapers ran successfully in order to be able to calculate the epu index for a day. How shall we store that information?

@peterdesmet @niconoe any ideas?

bartaelterman commented 9 years ago

Model JournalScraped added (c8c50972e48ea3b4acb278d2831c5466397c7b2b).
Added close_spider method to the pipeline, writing one line for every day and journal scraped.