Update sling assets - Githubissues

dagster-io / hooli-data-eng-pipelines

Example Dagster Cloud code for the Hooli Data Engineering organization.

72 stars 15 forks source link

Closed izzye84 closed 3 months ago

izzye84 commented 3 months ago

Updating the Sling demo to the new APIs.

Summary of updates:

added replication python config
updated SlingResource to use SlingConnectionResource instead of SlingSourceConnection/SlingTargetConnection (soon to be deprecated)
add custom sling translator to modify the default asset key (prefix with "RAW_DATA")
updated sling assets to use the sling_assets decorator

There seems to be a bug with local dev when including a target schema with duckdb — "Catalog Error: Schema with name "raw_data" already exists!"

github-actions[bot] commented 3 months ago

Your pull request at commit 6d047e86ce1d41e132992cc40c1c8f581170fd11 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
demo_assets	View in Cloud	May 17, 2024 at 01:10 AM (UTC)
basics	View in Cloud	May 17, 2024 at 01:10 AM (UTC)
batch_enrichment	View in Cloud	May 17, 2024 at 01:10 AM (UTC)
data-eng-pipeline	View in Cloud	May 17, 2024 at 01:10 AM (UTC)
snowflake_insights	View in Cloud	May 17, 2024 at 01:10 AM (UTC)

github-actions[bot] commented 3 months ago

Your pull request at commit e01c16d41ce1101b4abcdd1ebcdeb1fb81b8f907 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
demo_assets	Building...	May 17, 2024 at 03:28 AM (UTC)
basics	Building...	May 17, 2024 at 03:28 AM (UTC)
batch_enrichment	Building...	May 17, 2024 at 03:28 AM (UTC)
data-eng-pipeline	Building...	May 17, 2024 at 03:28 AM (UTC)
snowflake_insights	Building...	May 17, 2024 at 03:28 AM (UTC)

github-actions[bot] commented 3 months ago

Your pull request at commit 50569fd991e0e8433046e872d2b685063ad1e676 is automatically being deployed to Dagster Cloud.

Location	Link	Updated
demo_assets	View in Cloud	May 17, 2024 at 03:40 AM (UTC)
basics	View in Cloud	May 17, 2024 at 03:40 AM (UTC)
batch_enrichment	View in Cloud	May 17, 2024 at 03:40 AM (UTC)
data-eng-pipeline	View in Cloud	May 17, 2024 at 03:40 AM (UTC)
snowflake_insights	View in Cloud	May 17, 2024 at 03:40 AM (UTC)

github-actions[bot] commented 3 months ago

Your pull request at commit 3024396e28bc0bde640771e4110d36bf5562c43c is automatically being deployed to Dagster Cloud.

Location	Link	Updated
demo_assets	View in Cloud	May 24, 2024 at 03:22 PM (UTC)
basics	View in Cloud	May 24, 2024 at 03:22 PM (UTC)
batch_enrichment	View in Cloud	May 24, 2024 at 03:22 PM (UTC)
data-eng-pipeline	View in Cloud	May 24, 2024 at 03:22 PM (UTC)
snowflake_insights	View in Cloud	May 24, 2024 at 03:22 PM (UTC)