search

NYCPlanning / data-engineering

Primary repository for NYC DCP's Data Engineering team
14 stars 0 forks source link

GFT: Modify test for NYC Properties source #767

Closed sf-dcp closed 2 months ago

sf-dcp commented 2 months ago

Motivation

Nightly QA failed for Green Fast Track data product. A source data test failed for DPR Park Properties because new unexpected values in the typecategory column were detected. These new values are: Operations & Retired N/A.

Solution

Add the new values to the seed csv file with expected typecategory values. Successful build here

🚨 Action needed

@croswell81 & @jackrosacker, tagging both of you here. It seems that DPR Park Properties added new values in the typecategory column which we use for filtering records in GFT. These values are not in their data dictionary. Could you please confirm we should exclude DPR Park records with the values Operations & Retired N/A?

Edit: typecategory column, not subcategory

croswell81 commented 2 months ago

@sf-dcp First, confirming the field with the new values is 'typecategory' and not 'subcategory'?

If it is in subcategory, we can ignore. If it is in typecategory, I can pass along to Planning Support to ask DPR. Note, Planning Support indicated DPR may ask us to remove some of our values from our typecategory filter. Will pass along info when known.

cc: @jackrosacker

sf-dcp commented 2 months ago

@sf-dcp First, confirming the field with the new values is 'typecategory' and not 'subcategory'?

If it is in subcategory, we can ignore. If it is in typecategory, I can pass along to Planning Support to ask DPR. Note, Planning Support indicated DPR may ask us to remove some of our values from our typecategory filter. Will pass along info when known.

cc: @jackrosacker

Sorry, it's typecategory.

fvankrieken commented 2 months ago

Seems fine if you'd like to merge and simply make more changes when we get feedback, or also if you'd rather just keep this PR open so that it doesn't slip our minds

croswell81 commented 2 months ago

@sf-dcp First, confirming the field with the new values is 'typecategory' and not 'subcategory'? If it is in subcategory, we can ignore. If it is in typecategory, I can pass along to Planning Support to ask DPR. Note, Planning Support indicated DPR may ask us to remove some of our values from our typecategory filter. Will pass along info when known. cc: @jackrosacker

Sorry, it's typecategory.

@sf-dcp "Operations” & “Retired N/A” can be filtered out