KPDB for City of Yes for Housing Opportunity

NYCPlanning / data-engineering

Primary repository for NYC DCP's Data Engineering team

19 stars 0 forks source link

KPDB for City of Yes for Housing Opportunity #233

Open damonmcc opened 11 months ago

damonmcc commented 11 months ago

Project description

DCP has proposed a set of zoning text amendments called City of Yes for Housing Opportunity (CYHO). The Housing team must develop a new analytical model to do environmental review of CYHO. To do that, they need a version of KPDB where the dates of source data stop at or around 6/30/23.

Requirements

KPDB source data versions:
- ZAP data from mid-June or mid-July
- Housing DB 23Q2
Thorough QA by the Housing and Capital Planning teams
Fixes to any data issues flagged by QA

Timing

[x] #547
[x] the Housing team needs a version of KPDB where the dates of source data stops at or around 6/30/23, specifically:
- ZAP data from mid-June or mid-July
- Housing DB 23Q2
[x] GIS shares the map with Housing team during the week of 3/11
[ ] #665
[ ] review KPDB wiki for accuracy
[ ] https://github.com/NYCPlanning/edm-overview/issues/1009

Notes

New KPDB aggregation table Schema Doc from GIS

damonmcc commented 11 months ago

cc @AmandaDoyle

had issues with missing columns in ZAP data archived from June through July. used version 20230905 and confirmed with the Housing team that the date and date_type columns can be used to filter out unwanted records

generated outputs via this PR test run and will share with Housing Team when they choose which Sharepoint folder they'd like it in

damonmcc commented 10 months ago

built and shared with Housing team for QA on 10/3 via their Sharepoint folder

damonmcc commented 10 months ago

per closing of https://github.com/NYCPlanning/data-engineering/issues/252, must cherry pick or independently merge to main the fixes from #275

then using those fixes in this branch, must build and share with Housing team

damonmcc commented 8 months ago

next steps from 11/30

Damon sends HPD-related source files to Sam for revisions
Damon sends ZAP record ID correction file to Sam for revisions
- these revisions will be based on confirmations that certain old rezonings that are in ZAP shouldn't be in KPDB
Damon runs KPDB with:
- these and other updated corrections files
- changes to the code to move records with no geometry to their own table
- changes to the code to undo the duplicate records fix (since they may be caused by duplicate DOB records that the QATeam will correct)
QA Team reviews that new build with a focus on:
- corrected rezonings
- DOB records that have been assigned to multiple projects

damonmcc commented 8 months ago

build from 12/12 has issues with some corrections not being applied

damonmcc commented 7 months ago

build from 12/13 is in QA

damonmcc commented 7 months ago

corrections based on the 12/13/23 build were give to DE on 1/5/24 and used in build runs here

latest build is in QA

damonmcc commented 6 months ago

most urgent priority is to fix issues flagged by GIS in aggregation outputs

damonmcc commented 6 months ago

notes from Housing Map demo on 2/23

nice map
"future units without phasing" are pretty much completed units. by design, we haven't been dropping records with 100% completed units from KPDB so that people can see all units related to a project (past and future)
Housing Team would like to investigate records with net units and no phasing

damonmcc commented 5 months ago

awaiting new Data Dictionary from GIS. will use it to revise columns in KPDB aggregation tables (with a focus on the NTA and CDTA tables)