Add IDOT data to data-lake

ccao-data / data-architecture

Codebase for CCAO data infrastructure construction and management

https://ccao-data.github.io/data-architecture/

6 stars 4 forks source link

Closed Damonamajor closed 1 week ago

Damonamajor commented 1 month ago

This uploads the IDOT data from the following location to our data lake in order to process it into future characteristics.

The transformation first unions roads with identical characteristics into the same line string.
It then joins roads with the same name and missing features by averaging the features of intersecting roads with the same name.

wrridgeway commented 1 month ago

@Damonamajor don't forget to update dbt documentation with the new table

Damonamajor commented 1 month ago

@Damonamajor don't forget to update dbt documentation with the new table

I had it in the next branch, but happy to migrate it over.

wrridgeway commented 1 month ago

yeah, let's just for the spatial.traffic table.

couple other things for the warehouse script specifically:

let's recode 0000 in condition year to NA
look through the code book and see if any other data that should be NA needs to be re-coded
let's recode stuff like surface_type with the values their numbers correspond to, ie concrete rather than 800 for surface type, if we have that information.