search

os-climate / itr-data-pipeline

Produce data for ITR tool using data from Data Commons
Apache License 2.0
3 stars 4 forks source link

Ingest 2024 #76

Closed MichaelTiemannOSC closed 3 months ago

MichaelTiemannOSC commented 3 months ago

Restore rmi data sources to complete the verification that we can do an end-to-end ingestion with the new Notebooks. This is as far as we can test all the latest versions before we shift to the new data mesh pattern (and more modern versions of dbt and OpenMetadata).

MichaelTiemannOSC commented 3 months ago

There remains some disconnect between how dbt trims newlines and how the pre-commit linting wants there to be newlines, but this is a good checkpoint for switching across to the new data mesh pattern, and dealing with the newline problem at a later date.