23_adopted CPDB Update

[fvankrieken]

The purpose of this issue is to help the team track the tasks that need to occur before kicking off a CPDB update.

Update source data

Read more about who to reach out to and how to update a specific dataset here

Received from agency partner and manually loaded into data libraries

• [x] Capital Commitment Plan: fisa_capitalcommitments
• [x] ddc_capitalprojects_infrastructure
• [x] ddc_capitalprojects_publicbuildings
• [x] dot_projects_bridges
• [x] edc_capitalprojects - new data needs to be appended onto existing source dataset
• [x] edc_capitalprojects_ferry - new data is often not received and this dataset is not updated with each release because nothing has changed.

Automatically loaded into data libraries

Before initiating a build of cpdb, verify the latest version in DigitalOcean matches the version in NYC OpenData

Projects

• [x] cpdb_capital_spending - updated with action
• [x] dot_projects_intersections https://data.cityofnewyork.us/Transportation/Street-and-Highway-Capital-Reconstruction-Projects/97nd-ff3i
• [x] dot_projects_streets https://data.cityofnewyork.us/Transportation/Street-and-Highway-Capital-Reconstruction-Projects/jvk9-k4re
• [x] dpr_capitalprojects https://www.nycgovparks.org/bigapps
• [x] dpr_parksproperties https://data.cityofnewyork.us/Recreation/Parks-Properties/enfh-gkve
• [x] dcp_cpdb_agencyverified (does not get updated)

Building and lot-level info

• [x] dcp_mappluto_wi https://www1.nyc.gov/site/planning/data-maps/open-data/dwn-pluto-mappluto.page
• [x] dcp_facilities https://www1.nyc.gov/site/planning/data-maps/open-data/dwn-selfac.page
• [x] doitt_buildingfootprints https://data.cityofnewyork.us/Housing-Development/Building-Footprints/nqwf-w8eh

Spatial boundaries

Come from Geosupport update

• [x] dcp_stateassemblydistricts https://www1.nyc.gov/assets/planning/download/pdf/data-maps/open-data/nyad_metadata.pdf?ver=21d
• [x] dcp_ct2020 https://www1.nyc.gov/site/planning/data-maps/open-data/districts-download-metadata.page
• [x] dcp_congressionaldistricts https://www1.nyc.gov/site/planning/data-maps/open-data/districts-download-metadata.page
• [x] dcp_cdboundaries https://www1.nyc.gov/site/planning/data-maps/open-data/districts-download-metadata.page
• [x] dcp_statesenatedistricts https://www1.nyc.gov/site/planning/data-maps/open-data/districts-download-metadata.page
• [x] dcp_municipalcourtdistricts https://www1.nyc.gov/site/planning/data-maps/open-data/districts-download-metadata.page
• [x] dcp_school_districts https://www1.nyc.gov/site/planning/data-maps/open-data/districts-download-metadata.page
• [x] dcp_councildistricts https://www1.nyc.gov/assets/planning/download/pdf/data-maps/open-data/nycc_metadata.pdf

Other

• [x] dcp_trafficanalysiszones - this dataset is almost never updated
• [x] nypd_policeprecincts https://data.cityofnewyork.us/Public-Safety/Police-Precincts/78dh-3ptz
• [x] fdny_firecompanies https://data.cityofnewyork.us/Public-Safety/Fire-Companies/iiv7-jaj9

[damonmcc]

an initial build succeeded but I'm not sure all source data was updated for it. next steps seem to be:

• [ ] review source data
• [ ] run build
• [ ] review QAQC
• [ ] (optional) consider any tweaks to QAQC app

looking at CPDB QAQC app page:

• would like to sort the options in the "Select an export for comparison" dropdown so the latest/most recent are at the top
• love the "Key CPDB QAQC terms" section

[fvankrieken]

Hey sorry @damonmcc , was going to comment this morning. This issue is pretty up-to-date on data - I spent Friday mainly on the non-manual data stuff working through some things, the manual data still needs to be loaded

There's a PR in this repo but just to update issue template for something that was unclear to me

https://github.com/NYCPlanning/db-cpdb/pull/139

Looks like you already merged the data library PR which was tangientially related

https://github.com/NYCPlanning/db-data-library/pull/397

The initial successful build was before any data was updated

[AmandaDoyle]

@damonmcc - @fvankrieken is right. The source datasets that are not checked off need to be loaded into data library. These datasets came via email. If you can't find the emails, give me a shout. Once new data are loaded in, build CPDB with new data, then QAQC, and if things look send to capital planning. @fvankrieken - enjoy the desert!

[damonmcc]

@AmandaDoyle loaded data into data library and ran a build using branch 23_Q2_build (that branch has no code changes, but nice to not build on main yet)

from comparing to our 2023-03-10 output in the QAQC app, I see:

• no differences in the Total Number of Projects (concerning, looking into bad characters in inputs)
• increases in Percentages of Projects Mapped

[damonmcc]

currently making sure that

• source data is archived as latest
• github action for Archive Capital Spending Data is working
• build is working

[damonmcc]

the most recent build looks good, comparing latest output from 23_Q2_build branch to the latest on main (2023-03-10)

• increases in Total Number of Projects for all sponsoring agencies (max is 70)

next step is to upload outputs to Carto for Capital Planning team review

[damonmcc]

built on main and shared with Capital Planning for review