NYCPlanning / data-engineering

Primary repository for NYC DCP's Data Engineering team
14 stars 0 forks source link

GFT: update wpaa source and logic #762

Closed sf-dcp closed 1 month ago

sf-dcp commented 1 month ago

Per this comment, logic for wpaa variable needed additional twists. This PR addresses both "todo" bullet points in the comment.

The following has been implemented:

Note

  1. Old and new wpaa datasets have different geometry types. If I updated latest version with new data in recipes, it would yield issues in other ongoing PRs related to GFT. Therefore, I pulled new wpaa data without updating latest and fixed the wpaa version in GFT recipe file to work on this PR. Once PR is done:

    • [x] run data library to update latest version for wpaa dataset in recipes
    • [x] remove commit that fixes wpaa version in GFT recipe
  2. There is one failing pytest in dcpy/library. I didn't touch dcpy in this PR, so not sure what it is. The failure should be addressed in a different PR.

Successful run here

fvankrieken commented 1 month ago

Sorry for the flaky test - for testing live endpoints for ingest, should probably move that to the "nightly build" action instead of PRs

Hopefully rerunning clears it up for now though

sf-dcp commented 1 month ago

Sorry for the flaky test - for testing live endpoints for ingest, should probably move that to the "nightly build" action instead of PRs

Hopefully rerunning clears it up for now though

Yep, passes now!

sf-dcp commented 1 month ago

Re-ran data library for wpaa to update latest version in recipes, removed commit for wpaa version in GFT recipe, re-ran build with these changes (it passes here).