NYCPlanning / data-engineering

Primary repository for NYC DCP's Data Engineering team
25 stars 1 forks source link

Pluto 24v4 #1213

Open damonmcc opened 1 month ago

damonmcc commented 1 month ago

Main tasks

Data loading

Manual Updates

Updated 2x a year typically in June and December

Automated Updates

Open data automated pull

Check here to see the latest run

DOF Automated Pull and Number of Buildings

Updated with Quarterly updates (check here)

Updated with Zoning Taxlots

(check here for latest run).

These are all produced by GIS, who typically update them sometime in the first week of each month. Check in with them before archiving with data library

Never Updated (Safe to ignore)

damonmcc commented 3 weeks ago

will wait for Geosupport 24D (Linux) to be available on Bytes, expected soon after 11/18

also awaiting new Zoning data from GIS

caseysmithpgh commented 2 weeks ago

GIS ZTL QA Passed.

Latest Zoning and DTM available in staging. Once @jackrosacker is able to publish EDes to DO, PLUTO 24v4 is clear for build.

jackrosacker commented 2 weeks ago

GIS ZTL QA Passed.

Latest Zoning and DTM available in staging. Once @jackrosacker is able to publish EDes to DO, PLUTO 24v4 is clear for build.

E-Des has been published! PLUTO is good to build on our end.

sf-dcp commented 1 week ago

Researching "unreasonably small" condo bbls --> manually corrected 15 condo BBLs.

sf-dcp commented 1 week ago

PLUTO has been built and promoted to draft.

Notes on the QAQC page:

  1. The "source data" table has been moved up. DPR & OTI sources have been added to the table per GIS request.
  2. Outlier Analysis, Unreasonable Increase in Building Area: previously, this table incorrectly showed increase in lot area. Starting PLUTO 24v4, this table now shows building area as expected.

DE QA Notes:

I reviewed the reports where comparison type is "Last of same version type". Notes by QAQC page section:

1. Mismatched Records

2. Aggregate Changes

3. Expected Value Comparison

4. Outlier Analysis

5. New and Vanished BBLs