DevDB - issue with QAQC output

NYCPlanning / data-engineering

Primary repository for NYC DCP's Data Engineering team

20 stars 0 forks source link

DevDB - issue with QAQC output #151

Closed AmandaDoyle closed 1 year ago

AmandaDoyle commented 1 year ago

Housing reported that the b_nonres_with_units in the QAQC output does not seem to be flagging records correctly - too many records are outputted and when reading the description of the flag, the records it is flagging do not meet the criteria of being Non-Residential Jobs with Non-Zero Class A initial or proposed units. I have not dug in, but what is going on? It'd be great to figure this out early next week and SL will be back to collaborate with him.

fvankrieken commented 1 year ago

I can start looking into this tomorrow

fvankrieken commented 1 year ago

Just jotting down some quick notes for myself glancing out the outputs before looking at any code

Latest output in Final_qaqc.csv

10585 jobs with value 1
~254000 jobs with value 0
16958 with no value

22Q4

10563 jobs with value 1
~207000 jobs with value 0
37816 with no value

So number of jobs with no value has fallen by about 50%, but not sure if the numbers we were seeing in 22Q4 are accurate as well. But any of these numbers going down would be unexpected, right @AmandaDoyle ?

fvankrieken commented 1 year ago

Per SL, these normally aren't manually QC'd