search

wisemuffin / nsw-doe-data-stack-in-a-box

Department of Education (DOE) for New South Wales (AUS) data stack in a box
https://nsw-doe-data-stack-in-a-box.vercel.app
MIT License
23 stars 0 forks source link

dbt state modified (slim ci) with manifest json is not working properly #11

Open wisemuffin opened 4 months ago

wisemuffin commented 4 months ago

Manifest.json is not getting updated

This results in the dbt defer command not working correctly

image

need to setup dagster cli to run on push to main then to upload resulting manifest

wisemuffin commented 3 days ago

its caused by dbt state modified getting triggered on source changes between the database name e.g. changing "database": "nsw_doe_data_stack_in_a_box__dev" to "database": "nsw_doe_data_stack_in_a_box__prod" in the manifest is causing the issue.

Full section of the source that is where the issue is coming up:

"source.nsw_doe_data_stack_in_a_box.raw.raw_nsw_doe_datansw_class_size": {"database": "nsw_doe_data_stack_in_a_box__prod", "schema": "analytics", "name": "raw_nsw_doe_datansw_class_size", "resource_type": "source", "package_name": "nsw_doe_data_stack_in_a_box", "path": "models/staging/nsw_doe/sources.yml", "original_file_path": "models/staging/nsw_doe/sources.yml", "unique_id": "source.nsw_doe_data_stack_in_a_box.raw.raw_nsw_doe_datansw_class_size", "fqn": ["nsw_doe_data_stack_in_a_box", "staging", "nsw_doe", "raw", "raw_nsw_doe_datansw_class_size"], "source_name": "raw", "source_description": "", "loader": "", "identifier": "raw_nsw_doe_datansw_class_size", "quoting": {"database": null, "schema": null, "identifier": null, "column": null}, "loaded_at_field": null, "freshness": {"warn_after": {"count": null, "period": null}, "error_after": {"count": null, "period": null}, "filter": null}, "external": null, "description": "", "columns": {}, "meta": {"dagster": {"asset_key": ["raw__nsw_doe_datansw__class_size"]}}, "source_meta": {}, "tags": [], "config": {"enabled": true}, "patch_path": null, "unrendered_config": {}, "relation_name": "\"nsw_doe_data_stack_in_a_box__prod\".\"analytics\".\"raw_nsw_doe_datansw_class_size\"", "created_at": 1829114562.9676197}

sources are a top level object:

image

example of how to test:

dbt ls --select state:modified --state ./target/last_manifest --exclude "source:*"
wisemuffin commented 2 days ago

For now it works on just the changed models when using state:modified and --exclude "source:*".

But as soon as i add in + to `state:modified+ to run all upstream then it still references all models upstream of source: