Automate 10 datasets

[AcckiyGerman]

As a datahub user / data curator I want to have docs & examples of automated datasets so I'm able to automate my own dataset.

Acceptance criteria

• [ ] 10 datasets are automated and running
• [x] 'how to automate dataset' guide

tasks

• [x] take a list of ready-to-automate datasets here: https://github.com/datahq/pm/issues/84 or here: https://docs.google.com/spreadsheets/d/14kJluhePaMOx6vYBic0poVjDYK3I8-v_xF3sv4Focac/edit#gid=405792816

  for dataset in datasets:
  make flow.yaml
  check "data push-flow" works
  review_from_irakly

  Datasets to automate:

• [x] https://github.com/datasets/airport-codes
• [x] https://github.com/datasets/eu-emissions-trading-system
• [x] https://github.com/datasets/gdp-uk/
  • [95%] ready https://github.com/datasets/gdp-uk/issues/5 left
• [x] https://github.com/datasets/gold-prices/
  • [90%] ready, validation error with auto-inferred schema, see https://testing.datahub.io/core/gold-prices/v/11
• [ ] https://github.com/datasets/s-and-p-500 5 [60%] - can't remove last row, need 'Find/Replace'
  • lot of empty rows in xls file
  • date is formatted like this 2000.1 where 1 = October
• [ ] https://github.com/datasets/population [0%]
  • needs unpivoting
• [x] https://github.com/datasets/gdp [90%] ready
  • source file validation errors
• [ ] https://github.com/datasets/co2-ppm

• [ ] https://github.com/datasets/cpi

• [x] write notes on the way to create a guide then

Analyse

[zelima]

@AcckiyGerman please do not count gdp-uk as done. As we discussed we are not going to implement find/replace anytime soon, meaning it will never get to 100% in this sprint and get automated

[AcckiyGerman]

added some datasets that could be automated.

[AcckiyGerman]

datasets that probably could be automated after some efforts:

• cash-surplus-deficit https://github.com/datasets/cash-surplus-deficit
  • http://api.worldbank.org/indicator/GC.BAL.CASH.GD.ZS?format=csv
  • script is not working ATM
• co2-ppm https://github.com/datasets/co2-ppm
  • ftp://aftp.cmdl.noaa.gov/products/trends/co2/
  • lstrip
  • format-date
• cpi https://github.com/datasets/cpi
  • http://api.worldbank.org/indicator/FP.CPI.TOTL?format=csv
  • unpivot
  • remove header rows, add header
• cpi-us https://github.com/datasets/cpi-us
  • source is broken
  • unpivot
  • calculate new 'inflation' column
  • remove last rows
• gdp-us https://github.com/datasets/gdp-us
  • http://www.bea.gov/national/xls/gdplev.xls
  • format-date (quarters -> dates)
  • split 2 source into two resources
  • merge resources into 2 output files
• glacier-mass-balance https://github.com/datasets/glacier-mass-balance
  • http://www3.epa.gov/climatechange/images/indicator_downloads/glaciers_fig-1.csv
  • source is outdated (not updating)
• global-temp https://github.com/datasets/global-temp
  • source is broken
  • script is very complex :(
• global-temp-anomalies https://github.com/datasets/global-temp-anomalies
  • merge tables
  • complex formats normalizing
• natural-gas-prices https://github.com/datasets/natural-gas-prices
  • http://www.eia.gov/dnav/ng/hist_xls/RNGWHHDd.xls
  • format-date
• oil-prices https://github.com/datasets/oil-prices
  • source is broken
• smdg-master-terminal-facilities-list https://github.com/datasets/smdg-master-terminal-facilities-list
  • source is broken
• employment-us https://github.com/datasets/employment-us
  • source is broken
• expenditure-on-research-and-development https://github.com/datasets/expenditure-on-research-and-development
  • source is broken

[zelima]

Moving to backlog as we decided to hold automation for now,

[zelima]

WONTFIX/OBSOLETE