Update data products using `dcp_mappluto` to `dcp_mappluto_wi`

[mbh329]

Currently we have 3 MapPLUTO (dcp_mappluto, dcp_mappluto_wi, dcp_mappluto_clipped) related datasets that are ingested via data-library. In order to follow established patterns seen in our other datasets, we need to update dcp_mappluto to be the version that does not have water included boundaries (as of now, it is identical to dcp_mappluto_wi). To do this, we will need to change any instance in which one of our data products uses dcp_mappluto to dcp_mappluto_wi UNLESS otherewise specified...meaning..does the data product need to have the water included version or does it actually benefit from having a clipped version of the dataset. Each data product will have to carefully examined to determine the right version of MapPLUTO but generally they should include water as the larger boundaries help identify features that might not be included to a strictly land based boundary.

Data products/scripts using dcp_mappluto:

Current Repo's:

NYCPlanning/db-pluto

• [ ] pluto_build/sql/backfill.sql
• [ ] .github/workflows/publish.yml
• [ ] .github/workflows/publish_historical.yml
• [ ] Table mapping

NYCPlanning/db-data-library

• [ ] library/templates/dcp_mappluto.yml
• [ ] library/templates/dcp_mappluto_historical.yml
• [ ] library/templates/dcp_mappluto_clipped.yml
• [ ] tests/data/url.yml
• [ ] .github/workflows/single-runner.yml

NYCPlanning/db-facilities

• [ ] facdb/sql/_create_facdb_geom.sql
• [ ] facdb/sql/_create_facdb_spatial.sql
• [ ] facdb/bash/dataloading.sh
• [ ] facdb/cli.py
• [ ] /README.md

NYCPlanning/db-cpdb

• [ ] bash/01_dataloading.sh
• [ ] sql/attributes_agencyverified_geoms.sql
• [ ] .github/ISSUE_TEMPLATE/update.md
• [ ] Dataloading instructions
• [ ] Maintenance

NYCPlanning/db-developments

• [ ] sql/_geo.sql
• [ ] sql/_pluto.sql
• [ ] templates/dcp_developments.yml
• [ ] bash/01_dataloading.sh
• [ ] .github/ISSUE_TEMPLATE/update.md
• [ ] templates/dcp_housing.yml
• [ ] .github/workflows/test.yml
• [ ] sql/_function.sql
• [ ] 19Q4 Attribute mapping
• [ ] 20Q2 Data Dictionary
• [ ] Data Dictionary
• [ ] 20Q2 Attribute mapping
• [ ] Attribute mapping
• [ ] Description of tables

NYCPlanning/db-colp

• [ ] sql/create_colp.sql

NYCPlanning/action-library-archive

• [ ] .github/workflows/example.yml

LABS

NYCPlanning/labs-search-api

• [ ] search-helpers/bbl.js

NYCPlanning/labs-layers-api

• [ ] data/sources/pluto.json

NYCPlanning/labs-geosearch-pad-normalize

• [ ] /_download_data.R

NYCPlanning/labs-zola

• [ ] app/adapters/lot.js
• [ ] app/components/intersecting-layers.js
• [ ] tests/acceptance/visit-lot-test.js

NYCPlanning/labs-ember-search

• [ ] addon/components/labs-bbl-lookup.js

NYCPlanning/labs-applicantmaps

• [ ] app/components/project-geometries/modes/lots.js

NYCPlanning/labs-streets

• [ ] public/sources.json
• [ ] app/controllers/application.js
• [ ] public/v1/layer-groups

GIS

NYCPlanning/gis-mappluto-convert

• [ ] /MapPLUTOCSV2FC_Conversion.py

OLD/Deprecated

NYCPlanning/dob-permits-geocode

• [ ] scripts/geocode.sql

NYCPlanning/db-housing

• [ ] housing_build/sql/dcpgeoms.sql

NYCPlanning/db-facilities-old

• [ ] 3_geoprocessing/bbl2overwritegeom.sql
• [ ] 3_geoprocessing/setSRID_4326.sql
• [ ] /3_geoprocessing.sh
• [ ] 3_geoprocessing/vacuum.sql
• [ ] /oklog.txt
• [ ] 3_geoprocessing/join_PLUTOtabular.sql
• [ ] 3_geoprocessing/join_PLUTOspatial.sql
• [ ] /1_download.sh

NYCPlanning/action-library-archive

• [ ] README.md

NYCPlanning/db-civic-data-loader

• open_datasets/dcp_mappluto/after.sql
• open_datasets/dcp_mappluto/data.json

NYCPlanning/data-loading-scripts

• [ ] old/dcp_pluto/load.sh
• [ ] datasets/dcp_mappluto/data.json
• [ ] old/doitt_buildingfootprints/geocode.sql
• [ ] old/dcp_pluto/drop.sql
• [ ] old/dcp_pluto/updatetypes.sql

NYCPlanning/db-airflow-dags

• [ ] facdb_1_download/datasets/dcp_mappluto/after.sql
• [ ] facdb_3_geoprocessing/bbl2overwritegeom.sql
• [ ] facdb_1_download/datasets/dcp_mappluto/afterold.sql
• [ ] facdb_3_geoprocessing/setSRID_4326.sql
• [ ] facdb_3_geoprocessing/vacuum.sql
• [ ] /facdb_3_geoprocessing.py
• [ ] facdb_1_download/datasets/doe_facilities_lcgms/after.sql
• [ ] facdb_1_download/datasets/dcp_mappluto/data.json
• [ ] facdb_2_assembly/create_bblbin_one2one.sql
• [ ] - [ ] facdb_3_geoprocessing/join_PLUTOtabular.sql
• [ ] facdb_3_geoprocessing/join_PLUTOspatial.sql
• [ ] /facdb_1_download.py

NYCPlanning/db-data-recipes

• [ ] archive/dcp_mappluto/build.py

actions-marketplace-validations/NYCPlanning_action-library-archive

• [ ] /dcp_mappluto.yml
• [ ] .github/workflows/example.yml

NYCPlanning/db-pad

• [ ] /building_composite.sql

[AmandaDoyle]

@mbh329 Thanks for putting this together. This is quite far reaching. Based on this we cannot change dcp_mappluto in data library to point to the clipped version of mappluto anytime soon. Here's what I see as the logical steps to update dcp_mappluto to dcp_mappluto_wi to be in line with other datasets in data lib: 1) Update DE repos where dcp_mappluto is being pulled in via bash/dataloading.sh step, which looks like 3 repos: 1) db-facilities, db-cpdb, and db-developments 2) Investigate where / how OSE and GIS are pulling dcp_mappluto from and what would the implications be if we update data library? 3) Investigate other DE repos where dcp_mappluto is mentioned, but not via a dataloading step and determine what needs to change and map out any downstream impacts. 4) Ignore any repo that's archived. We can start on step 1 as a maintenance task whenever we want to, and we can schedule as a project. 2 and 3 may take a little more time. What do you think?