search

CodeForAfrica-SCRAPERS / healthtools_ke

[morph] Scrapers for the HealthTools Kenya data.
https://morph.io/CodeForAfrica-SCRAPERS/healthtools_ke
MIT License
3 stars 15 forks source link

Store stats of scraping #41

Open DavidLemayian opened 7 years ago

DavidLemayian commented 7 years ago

After scraping, we should store stats in s3 in a stats.json file that we will be used in display on a HTML page. This should include:

  1. Count of each data scraped.
  2. Dates of last successfully scraped for each and date of last successful scrape as a whole.
  3. How long each scrape took and all the scrapers.

For Debate: This info should also be pushed to Google Analytics at a later stage.

DavidLemayian commented 6 years ago

Are there any stats services / API / storage that we can benchmark off of?