[build] version: Major: 23v1

[AmandaDoyle]

Update version name

PLUTO now uses version numbering YYvMAJOR.MINOR

1. YY for the last two digits of the release year
2. MAJOR version for using the latest versions of all input data
3. MINOR version for using the latest versions of particular input data

Major Release: To initiate a major release, you must change the VERSION and VERSION_PREV. All other variables can remain unchanged and will default to latest in the dataloading step.

• [x] https://github.com/NYCPlanning/db-pluto/blob/main/pluto_build/version.env

Minor Release: To initiate a minor release, you must change the VERSION and VERSION_PREV. In addition, you are required to hold the following variables constant with the last major release of PLUTO (you can reference the source_data_version table):

• [ ] https://github.com/NYCPlanning/db-pluto/blob/main/pluto_build/version.env

• [ ] DOF_WEEKLY_DATA_VERSION

• [ ] DOF_CAMA_DATA_VERSION

• [ ] GEOSUPPORT_VERSION

• [ ] FEMA_FIRPS_VERSION

• [ ] DOITT_DATA_VERSION

• [ ] DOF_DATA_VERSION

• [ ] DCP_COLP_VERSION

• [ ] DPR_GREENTHUMB_VERSION

• [ ] DSNY_FREQUENCIES_VERSION

• [ ] LPC_HISTORIC_DISTRICTS_VERSION

• [ ] LPC_LANDMARKS_VESRSION

• [ ] PLUTO_CORRECTIONS_VERSION

  Data loading

Manual Updates

Updated 2x a year typically in June and December

• [x] dcp_colp (check here)

• [ ] Currently updated with each new release of a major version of PLUTO - important to make sure this is up to date pluto_corrections (pulling from bytes, must update when there's updates to pluto_input_research) This is no longer relevant with updated process

Automated Updates

Open data automated pull

Check here to see the latest run

• [x] dsny_frequencies
• [x] dpr_greenthumb
• [x] lpc_historic_districts
• [x] lpc_landmarks

DOF Automated Pull and Number of Buildings

• [x] pluto_pts (Check here)
• [x] pluto_input_geocodes (Check here)
• [x] pluto_input_cama_dof (Check here)
• [x] pluto_input_numbldgs (Check here)

Updated with Quarterly updates (check here)

• [x] dcp_cdboundaries_wi
• [x] dcp_ct2010
• [x] dcp_cb2010
• [x] dcp_ct2020
• [x] dcp_cb2020
• [x] dcp_school_districts
• [x] dcp_councildistricts_wi
• [x] dcp_firecompanies
• [x] dcp_policeprecincts
• [x] dcp_healthareas
• [x] dcp_healthcenters

Updated with Zoning Taxlots (check here for latest run). Note: for a minor release of PLUTO, these will be the only datasets that will be updated.

• [x] dof_dtm
• [x] dof_shoreline
• [x] dof_condo
• [x] dcp_commercialoverlay
• [x] dcp_limitedheight
• [x] dcp_zoningdistricts
• [x] dcp_specialpurpose
• [x] dcp_specialpurposesubdistricts
• [x] dcp_zoningmapamendments
• [x] dcp_edesignation

Never Updated (Safe to ignore)

• [x] doitt_zipcodeboundaries (almost never updated, check here)
• [x] fema_firms2007_100yr
• [x] fema_pfirms2015_100yr
• [x] dcp_zoningmapindex

Update QAQC App

- [ ] update data-engineering-qaqc version_comparison_report in src/pluto/pluto.py with version of pluto build (must be updated with every version of pluto [here] (https://github.com/NYCPlanning/data-engineering-qaqc/blob/main/src/pluto/pluto.py))

PLUTO Improvements (corrections)

Build CI Runs

• build on dev branch
• build on main branch

Comments

[damonmcc]

re builds failing on branch 23v1-dev

• the get_version bash function doesn't like the string 22v3.1, specifically curl doesn't like it
• I thought this was due to the period in the string, but it seems to be that the data isn't in edm-recipes
• archiving 22v3.1 isn't straightforward because it's published to a URL using 22v3_1
• working on it

[damonmcc]

archive of 22v3.1 worked by using an input version of 22v3_1. started a build with underscore version notation

[damonmcc]

@AmandaDoyle build succeeded but use of QAQC for this new version number isn't possible until this new version number is explicitly listed in the app's code

will start on a fix for the QAQC app, but manual inspection of BCTCB2020 values related to issue #432 can be done on data in DO

primarily using existing relevant issue https://github.com/NYCPlanning/data-engineering-qaqc/issues/249

[damonmcc]

the QAQC app is now able to view this version of data, but it hasn't been run on main and CAMA data is still unavailable

started draft PR https://github.com/NYCPlanning/db-pluto/pull/437 to prepare for merge to/build on main

[AmandaDoyle]

AD Notes on PLUTO QAQC: 1) Use QAQC app to identify any anomalies. Use descriptions as guides. 2) See if BCTCB2020 being NULL is still an issue #432 3) Make sure research is being applied #434 and the updated process for applying research is working as expected #424 4)Make sure city council district values are not null and the "current" is being applied #430 5) Make sure condos have geometries #410

Once CAMA is released at the end of this week we'll want to kick off another build; ideally by the end of this week before updated PTS data is pulled in over the weekend.

[damonmcc]

having

• fixed DOF DTM issue to produce ZTLDB (https://github.com/NYCPlanning/db-zoningtaxlots/issues/130)
• re-uploaded the file to ensure the projection is as expected
• ensured Digital Ocean permissions on it are public

a build on the dev branch is underway

[damonmcc]

added links to builds on dev and main branches to issue description