Identity 10-15 more datasets for automation

[AcckiyGerman]

Epic: https://github.com/datahq/pm/issues/78

As a code curator I want to know which core datasets could be automated, so I could automate them. As a datopian developer I want to know which of the possible new features are we going to implement next, so I'm doing the relevant work efforts.

Acceptance criteria:

list of 10 ready to automate datasets

• gdp https://github.com/datasets/gdp
  • http://api.worldbank.org/indicator/NY.GDP.MKTP.CD?format=csv
  • unpivot
  • auto-schema
• gdp-uk https://github.com/datasets/gdp-uk
  • https://www.ons.gov.uk/generator?format=csv&uri=/economy/grossdomesticproductgdp/timeseries/abmi/qna
  • date-format (quarters -> dates)
• gold-prices https://github.com/datasets/gold-prices
  • http://www.bundesbank.de/cae/servlet/StatisticDownload?tsId=BBEX3.M.XAU.USD.EA.AC.C06&its_csvFormat=en&its_fileFormat=csv&mode=its
  • format-date
• s-and-p-500 https://github.com/datasets/s-and-p-500
  • http://www.econ.yale.edu/~shiller/data/ie_data.xls
  • normalise date format
  • normalise formats

There is no 10 of 'ready to automate' datasets

list of datasets, that probably could be automated

Datasets that need some features, that we don't have ATM, or datasets, that are broken, and needs to be fixed befor automation:

• cash-surplus-deficit https://github.com/datasets/cash-surplus-deficit
  • http://api.worldbank.org/indicator/GC.BAL.CASH.GD.ZS?format=csv
  • script is not working ATM
• co2-ppm https://github.com/datasets/co2-ppm
  • ftp://aftp.cmdl.noaa.gov/products/trends/co2/
  • lstrip
  • format-date
• cpi https://github.com/datasets/cpi
  • http://api.worldbank.org/indicator/FP.CPI.TOTL?format=csv
  • unpivot
  • remove header rows, add header
• cpi-us https://github.com/datasets/cpi-us
  • source is broken
  • unpivot
  • calculate new 'inflation' column
  • remove last rows
• gdp-us https://github.com/datasets/gdp-us
  • http://www.bea.gov/national/xls/gdplev.xls
  • format-date (quarters -> dates)
  • split 2 source into two resources
  • merge resources into 2 output files
• glacier-mass-balance https://github.com/datasets/glacier-mass-balance
  • http://www3.epa.gov/climatechange/images/indicator_downloads/glaciers_fig-1.csv
  • source is outdated (not updating)
• global-temp https://github.com/datasets/global-temp
  • source is broken
  • script is very complex :(
• global-temp-anomalies https://github.com/datasets/global-temp-anomalies
  • merge tables
  • complex formats normalizing
• natural-gas-prices https://github.com/datasets/natural-gas-prices
  • http://www.eia.gov/dnav/ng/hist_xls/RNGWHHDd.xls
  • format-date
• oil-prices https://github.com/datasets/oil-prices
  • source is broken
• smdg-master-terminal-facilities-list https://github.com/datasets/smdg-master-terminal-facilities-list
  • source is broken
• employment-us https://github.com/datasets/employment-us
  • source is broken
• expenditure-on-research-and-development https://github.com/datasets/expenditure-on-research-and-development
  • source is broken

tasks

• [x] check core list and chose 10 datasets which could be automated now [5h]
• [x] list of datasets that could be automated with minimal efforts (after adding small features) [5h]

Core list to work with: https://docs.google.com/spreadsheets/d/14kJluhePaMOx6vYBic0poVjDYK3I8-v_xF3sv4Focac/edit#gid=405792816

[rufuspollock]

@zelima have you reviewed this? Isn't this a duplicate of the work we already did in #60? /cc @Mikanebu

[zelima]

@rufuspollock there are only 5 in #60. We want 10 ready and 15 new here.

@AcckiyGerman please do not paste hack.md links in GitHub issues (I removed now)

[rufuspollock]

@zelima got it. The title of issue is maybe misleading then because we already did the sorting (in theory). Maybe retitle to: identity next 10-15 datasets for automation.

[AcckiyGerman]

FIXED I looked through all datasets listed in our 'core data analysis' doc. Output:

• list of missed features
• datasets are flagged by 'could be automated': [yes, no, probably, broken source]

Output is in the same doc and the datahub analysis hackmd doc.

[zelima]

@AcckiyGerman please list those 10-15 datasets here before closing so we are all clear and do not have to open additional links (I would add here ones with "probably" and "source is broken"). In the following structure

• datasets name - link to github
  • link to source if possible
  • features needed if possible

[AcckiyGerman]

@zelima updated the lists in the issue:

• 4 datasets is ready to be automated (there were more, but some are already automated)
• 13 datasets could be automated after fixing sources or adding extra features to pipeline

[zelima]

This is great! we can close this now as FIXED