NYCPlanning / data-engineering

Primary repository for NYC DCP's Data Engineering team
23 stars 0 forks source link

create dbt tables for ZTL #936

Closed HengJiang0206 closed 4 months ago

HengJiang0206 commented 4 months ago

resolves #925

Creates stg tables for ZTL, dbt build completed successfully:

Screenshot 2024-06-21 at 13 58 06

Creates preliminary int tables for ZTL, dbt build completed successfully:

Screenshot 2024-06-21 at 17 36 25

Create preliminary SQL queries for the product table. dbt build completed successfully. Screenshot 2024-06-24 at 17 17 58

** Modifications made to product table and int tables The primary key for the product table is dtm_id. _intermediate_models.yml was modified to add table expectations that dtm_id or the combination of dtm_id and the joined column ids are unique, avoiding duplicate records. When pivoting the int tables using row_numbers, corrections were made to the queries to ensure that the joining process doesn't create duplicate dtm_id. The product table passed all tests and has dtm_id as its unique identifier.