search

dagster-io / hooli-data-eng-pipelines

Example Dagster Cloud code for the Hooli Data Engineering organization.
75 stars 15 forks source link

make amps work better for hooli #49

Closed slopp closed 1 year ago

slopp commented 1 year ago

This PR addresses some long standing problem's in Hooli's use of experimental auto-materialization policies, specifically the case of daily partitioned assets -> weekly partitioned assets -> unpartitioned assets.

The customized policy now makes it so that any change to a daily partition will cause downstream assets to materialize, essentially making AMPs more aggressive. This change helps us demo how changes propagate through the system.

We also remove lazy policies which just didn't work in these situations.

Tradeoffs:

The remaining freshness policies are now useful for alerting only, they no longer dictate run behavior.

The change to use a customized AMP means the AMP policies must be defined in code and not in the dbt model config.

github-actions[bot] commented 1 year ago

Your pull request is automatically being deployed to Dagster Cloud.

Location Status Link Updated
batch_enrichment View in Cloud Oct 24, 2023 at 07:38 PM (UTC)
data-eng-pipeline View in Cloud Oct 24, 2023 at 07:38 PM (UTC)
basics View in Cloud Oct 24, 2023 at 07:38 PM (UTC)