Adding dbt slim CI job - Githubissues

dagster-io / hooli-data-eng-pipelines

Example Dagster Cloud code for the Hooli Data Engineering organization.

72 stars 15 forks source link

Adding dbt slim CI job #60

Closed izzye84 closed 7 months ago

izzye84 commented 7 months ago

Adding dbt slim CI functionality.

High level process:

Copy production manifest.json to S3
Created @op and @job to use dbt's state deferral syntax (e.g., dbt build --select state:modified+ --defer --state <path/to/prod_manifest.json> ), since assets don't currently support it
GitHub Action:
- Reads production manifest.json from S3
- Generates new manifest.json with code changes
- Triggers op job above using the two manifests for comparison and only materialize new dbt models + downstream dependencies

github-actions[bot] commented 7 months ago

Your pull request is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
snowflake_insights		View in Cloud	Jan 19, 2024 at 06:46 PM (UTC)
demo_assets		View in Cloud	Jan 19, 2024 at 06:46 PM (UTC)
data-eng-pipeline		View in Cloud	Jan 19, 2024 at 06:46 PM (UTC)
basics		View in Cloud	Jan 19, 2024 at 06:46 PM (UTC)
batch_enrichment		View in Cloud	Jan 19, 2024 at 06:46 PM (UTC)