search

NYCPlanning / db-factfinder

data ETL for population fact finder (decennial + acs)
https://nycplanning.github.io/db-factfinder/factfinder/
MIT License
2 stars 3 forks source link

adding acs 2006-2010 metadata mapping, received from erica #171

Closed SPTKL closed 3 years ago

SPTKL commented 3 years ago

this should address #153

Script used to generate this file:

import pandas as pd
import requests
import ast
import json

df = pd.read_csv('meta2010.csv')
df['census_variable'] = df['census_variable'].apply(ast.literal_eval)
df['rounding'] = df['rounding'].fillna('0').astype(int)
df['source'] = ''
df = df.loc[df.Notes != 'remove', ['pff_variable','census_variable', 'base_variable', 'domain', 'rounding', 'source']]
df_dict = df.to_dict('records')
with open('metadata.json', 'w') as f:
    json.dump(df_dict, f, indent=4)

Running in github actions to test out results: https://github.com/NYCPlanning/db-factfinder/actions/runs/1052808059

SPTKL commented 3 years ago

Seems like we have a few failures:

For the poverty variables, we would need to create the {{ variable }}_pct variables too ref: #72 e.g. https://github.com/NYCPlanning/db-factfinder/issues/72#issuecomment-759574868 image

⛔️ FAILURE: pbwpv city ⛔️ FAILURE: pbwpv borough

⛔️ FAILURE: pu18pvu NTA ⛔️ FAILURE: pu18pvu CDTA ⛔️ FAILURE: pu18pvu CT20 ⛔️ FAILURE: pu18pvu city ⛔️ FAILURE: pu18pvu borough

⛔️ FAILURE: smpntc NTA ⛔️ FAILURE: smpntc CDTA ⛔️ FAILURE: smpntc CT20 ✅ SUCCESS: smpntc city ✅ SUCCESS: smpntc borough

⛔️ FAILURE: avgfmsz NTA ⛔️ FAILURE: avgfmsz CDTA ⛔️ FAILURE: avgfmsz CT20 ✅ SUCCESS: avgfmsz city ✅ SUCCESS: avgfmsz borough

⛔️ FAILURE: mafr NTA ⛔️ FAILURE: mafr CDTA ⛔️ FAILURE: mafr CT20 ⛔️ FAILURE: mafr city ⛔️ FAILURE: mafr borough

⛔️ FAILURE: norway NTA ⛔️ FAILURE: norway CDTA ⛔️ FAILURE: norway CT20 ⛔️ FAILURE: norway city ⛔️ FAILURE: norway borough

⛔️ FAILURE: grpintc NTA ⛔️ FAILURE: grpintc CDTA ⛔️ FAILURE: grpintc CT20 ✅ SUCCESS: grpintc city ✅ SUCCESS: grpintc borough

⛔️ FAILURE: popinfms NTA ⛔️ FAILURE: popinfms CDTA ⛔️ FAILURE: popinfms CT20 ✅ SUCCESS: popinfms city ✅ SUCCESS: popinfms borough

✅ SUCCESS: p65plbwpv NTA ✅ SUCCESS: p65plbwpv CDTA ✅ SUCCESS: p65plbwpv CT20 ⛔️ FAILURE: p65plbwpv city ⛔️ FAILURE: p65plbwpv borough

⛔️ FAILURE: pu18bwpv NTA ⛔️ FAILURE: pu18bwpv CDTA ⛔️ FAILURE: pu18bwpv CT20 ⛔️ FAILURE: pu18bwpv city ⛔️ FAILURE: pu18bwpv borough

⛔️ FAILURE: cameroon NTA ⛔️ FAILURE: cameroon CDTA ⛔️ FAILURE: cameroon CT20 ⛔️ FAILURE: cameroon city ⛔️ FAILURE: cameroon borough

⛔️ FAILURE: sweden NTA ⛔️ FAILURE: sweden CDTA ⛔️ FAILURE: sweden CT20 ⛔️ FAILURE: sweden city ⛔️ FAILURE: sweden borough

⛔️ FAILURE: fam3 NTA ⛔️ FAILURE: fam3 CDTA ⛔️ FAILURE: fam3 CT20 ✅ SUCCESS: fam3 city ✅ SUCCESS: fam3 borough

⛔️ FAILURE: nmsmpntc NTA ⛔️ FAILURE: nmsmpntc CDTA ⛔️ FAILURE: nmsmpntc CT20 ✅ SUCCESS: nmsmpntc city ✅ SUCCESS: nmsmpntc borough

⛔️ FAILURE: omafr NTA ⛔️ FAILURE: omafr CDTA ⛔️ FAILURE: omafr CT20 ⛔️ FAILURE: omafr city ⛔️ FAILURE: omafr borough

SPTKL commented 3 years ago

merging for now, and issues will be addressed separately in a different PR