snowplow / dbt-snowplow-utils

Snowplow utility functions to be used in conjunction with the snowplow-web dbt package.
Other
13 stars 6 forks source link

Add warning when no data is going to be processed #109

Closed rlh1994 closed 1 year ago

rlh1994 commented 1 year ago

Description & motivation

This adds a warning when the return_limits_from_model macro has not found any limits to return. This happens (mostly, looking at web) for the sessions_this_run table to get the events_this_run table, but also seems to be used later in consent, and I assume in other packages as well.

When it matches this condition, the warning is raised and the limits are set to the year 9999, this is to avoid a failure for BQ when the table requires a partition filter (see https://github.com/snowplow/dbt-snowplow-web/issues/129), so that now all warehouses should finish their run but with a warning.

Currently raising a warning doesn't actually give the model a WARN status *(https://github.com/dbt-labs/dbt-core/issues/6721) so I have padded the message so it's a bit more obvious.

I don't forsee this breaking anything, it only happens when there is no new data and the dates are so far in the future unless that field has a default to avoid being null (which most warehouses will use 9999-12-31 anyway then the new dates inserted should also not make a differnece

Checklist