Add schema checks - Githubissues

dagster-io / hooli-data-eng-pipelines

Example Dagster Cloud code for the Hooli Data Engineering organization.

72 stars 15 forks source link

Add schema checks #89

Closed cnolanminich closed 3 months ago

cnolanminich commented 4 months ago

This PR does a two things:

boring upkeep
- bumps Dagster version to 1.7.5
- fixes freshness checks to use the new list return value present in 1.7.5
- pins dbt-core to 1.7.x since 1.8 was released yesterday and breaks our build until dagster-dbt is upgraded
(the main event!) adds schema checks

Schema checks:

This PR currently adds schema checks to two raw assets and all dbt assets. Points for feedback:

is it bad to add a build_column_schema_change_checks to the definitions.py module? Is there any weird overhead / issues to think about?
this works locally, will mark as ready for review once next Dagster release is out.

Pre:

After casting order_date to a string 🎉

github-actions[bot] commented 4 months ago

Your pull request at commit e180a8ffdd69d6ac73b7ed3217d7233e5519b631 is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
demo_assets		View in Cloud	May 01, 2024 at 02:18 PM (UTC)
batch_enrichment		View in Cloud	May 01, 2024 at 02:18 PM (UTC)
data-eng-pipeline		View in Cloud	May 01, 2024 at 02:18 PM (UTC)
basics		View in Cloud	May 01, 2024 at 02:18 PM (UTC)
snowflake_insights		View in Cloud	May 01, 2024 at 02:18 PM (UTC)

github-actions[bot] commented 4 months ago

Your pull request at commit 23f6e3e1fb79b9e30f34015909725358084ba6c8 is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
demo_assets		Deploy failed	May 01, 2024 at 09:17 PM (UTC)
batch_enrichment		Deploy failed	May 01, 2024 at 09:17 PM (UTC)
data-eng-pipeline		Deploy failed	May 01, 2024 at 09:17 PM (UTC)
basics		Deploy failed	May 01, 2024 at 09:17 PM (UTC)
snowflake_insights		Deploy failed	May 01, 2024 at 09:17 PM (UTC)

github-actions[bot] commented 4 months ago

Your pull request at commit 1487d57666dc08f84f6b4d3e84c96145c5dc22a2 is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
demo_assets		Deploy failed	May 03, 2024 at 12:04 AM (UTC)
batch_enrichment		Deploy failed	May 03, 2024 at 12:04 AM (UTC)
data-eng-pipeline		Deploy failed	May 03, 2024 at 12:04 AM (UTC)
basics		Deploy failed	May 03, 2024 at 12:04 AM (UTC)
snowflake_insights		Deploy failed	May 03, 2024 at 12:04 AM (UTC)

github-actions[bot] commented 4 months ago

Your pull request at commit 589ef3d0a5cd65e2e6085e6073d4d47a37b3c63e is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
demo_assets		Building...	May 09, 2024 at 06:56 PM (UTC)
batch_enrichment		Building...	May 09, 2024 at 06:56 PM (UTC)
data-eng-pipeline		Building...	May 09, 2024 at 06:56 PM (UTC)
basics		Building...	May 09, 2024 at 06:56 PM (UTC)
snowflake_insights		Building...	May 09, 2024 at 06:56 PM (UTC)

github-actions[bot] commented 4 months ago

Your pull request at commit 18b9bb61504e16e849be403e7bf8d2b1143bb374 is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
demo_assets		Building...	May 09, 2024 at 07:04 PM (UTC)
batch_enrichment		Building...	May 09, 2024 at 07:04 PM (UTC)
data-eng-pipeline		Building...	May 09, 2024 at 07:04 PM (UTC)
basics		Building...	May 09, 2024 at 07:04 PM (UTC)
snowflake_insights		Building...	May 09, 2024 at 07:04 PM (UTC)

github-actions[bot] commented 4 months ago

Your pull request at commit b2b2d094478740bec1e26bdf8019211f926198bc is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
demo_assets		Building...	May 09, 2024 at 07:12 PM (UTC)
batch_enrichment		Building...	May 09, 2024 at 07:12 PM (UTC)
data-eng-pipeline		Building...	May 09, 2024 at 07:12 PM (UTC)
basics		Building...	May 09, 2024 at 07:12 PM (UTC)
snowflake_insights		Building...	May 09, 2024 at 07:12 PM (UTC)

github-actions[bot] commented 4 months ago

Your pull request at commit 089a93e8fd8810d1b75c1b37b95f27dd9cb5e3de is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
demo_assets		Deploy failed	May 09, 2024 at 07:24 PM (UTC)
batch_enrichment		Deploy failed	May 09, 2024 at 07:24 PM (UTC)
data-eng-pipeline		Deploy failed	May 09, 2024 at 07:24 PM (UTC)
basics		Deploy failed	May 09, 2024 at 07:24 PM (UTC)
snowflake_insights		Deploy failed	May 09, 2024 at 07:24 PM (UTC)

github-actions[bot] commented 3 months ago

Your pull request at commit b886fe62623e3fb5bddbcb28e47f4a01f205569b is automatically being deployed to Dagster Cloud.

Location	Status	Link	Updated
demo_assets		View in Cloud	May 10, 2024 at 01:51 PM (UTC)
batch_enrichment		View in Cloud	May 10, 2024 at 01:51 PM (UTC)
data-eng-pipeline		View in Cloud	May 10, 2024 at 01:51 PM (UTC)
basics		View in Cloud	May 10, 2024 at 01:51 PM (UTC)
snowflake_insights		View in Cloud	May 10, 2024 at 01:51 PM (UTC)