NYCPlanning / data-engineering

Primary repository for NYC DCP's Data Engineering team
14 stars 0 forks source link

FacDB 24v1: address issues from QAQC (round 2) #849

Closed sf-dcp closed 1 month ago

sf-dcp commented 1 month ago

Related to #461.

GIS found 2 new issues as a part of their 2nd QAQC (original issues from 1st QAQC review were already addressed):

1. nysdec_lands

In both 24v1 draft builds, the qaqc report was showing zero nysdec_lands facilities when compared to 88 facilities in 23v2 version. After inspecting the source data, it seems that the csv obtained from the source website had all records with null geometries resulting in zero nysdec_lands records in facdb table: image

Solution: instead of downloading a csv from the website, update nysdec_lands recipe to pull data from arcgis server. New data now shows geometries:

image

Successful pull of updated recipe here.

2. dot_pedplazas

The facdb table didn't have a single dot_pedplazas facility. In previous builds, we received a dot_pedplazas file from the DOT. I updated the dot_pedpazas recipe to pull data from Socrata but didn't update the source projection, incorrectly mapping dot_pedplazas facilities in a different state.

Solution: I updated the projection and now the count has been restored:

image

Successful build here.

Note, I plan to remove last commit clipping recipe versions prior to merging to main.