NYCPlanning / db-factfinder

data ETL for population fact finder (decennial + acs)
https://nycplanning.github.io/db-factfinder/factfinder/
MIT License
2 stars 3 forks source link

adding special outlier removal process for city/borough level #217

Closed SPTKL closed 3 years ago

SPTKL commented 3 years ago

controlled variables so that cv is not null

this is only the case for NYC and boroughs, in the data download these instances would appear as *****, it seems like in the API it is appearing in this case as "-555555555". This means the estimate is controlled. Rather than keeping the MOE null, we decided to show it as 0. This seems to be working sometimes in your code because the MOEs and percent MOEs are appearing as 0, but the CV is not calculated for some of these cases. I'm not sure the implementation step, but this issue of controlled estimates is only for NYC and boroughs.

Context

CV is missing for ACS data where MOE=0 for NYC and boroughs. CV should equal 0. (Row should not be gray)

Check data: 2015-2019 PopU181: Brooklyn & Queens Pop65pl: Queens Pop65plM: Queens Pop65plF: Queens Hsp1: NYC and all boroughs NHsp: NYC and all boroughs Pop_1: NYC and all boroughs-- ok PopU18_2: Brooklyn & Queens-- ok Pop65pl2: Queens-- ok Pop_2: NYC and all boroughs-- ok Hsp2: NYC and all boroughs-- ok 2006-2010 PopU5: Queens Pop65pl1: Queens Pop65plM: Queens Pop65plF: Queens Hsp1: NYC and all boroughs NHsp: NYC and all boroughs Pop_1: NYC and all boroughs-- ok Pop25t29: Queens-- ok Pop30t34: Queens-- ok Pop45t49: Queens-- ok Pop50t54: Brooklyn, Manhattan, Queens-- ok Pop65pl2: Queens-- ok Pop_2: NYC and all boroughs-- ok Hsp2: NYC and all boroughs-- ok

https://dev.azure.com/NYCPlanning/cc280b0d-40a0-4689-b852-2e6247f1af50/_apis/wit/attachments/4084dd3a-de6e-4b74-949d-b17a98092e1a?fileName=image.png

AB#5191