MattTriano / analytics_data_where_house

An analytics engineering sandbox focusing on real estates prices in Cook County, IL
https://docs.analytics-data-where-house.dev/
GNU Affero General Public License v3.0
9 stars 0 forks source link

Configure dbt's warn-error parameter to ignore some warnings #82

Closed MattTriano closed 1 year ago

MattTriano commented 1 year ago

dbt's warn-error option causes warnings to be treated as errors, and I added this option so that the user is forced to see these warnings. But while testing a fresh setup (in my distributed_data_where_house skeleton project), I discovered that dbt will raise warnings if there aren't any dbt nodes (aka "resources"; includes models) in paths defined in the dbt_project.yml file.

As I built the pipeline-update taskflow to automatically generate dbt models as the DAG runs, a fresh setup (that drops all of the included models) would always define paths to the dbt/models/ directories data_raw, clean, feature, dwh, and report but would only have model files in the data_raw dir at the time it would reach the first dbt-model-running script.

Per the dbt v1.4 changelog, it looks like it's possible to define which warnings should be raised as errors. Determine how to configure dbt to ignore the issue above (ideally tailored as narrowly as possible, so that other defined-but-unused paths still raise errors and get attention).

MattTriano commented 1 year ago

Reference material of note: