Closed izzye84 closed 3 months ago
Your pull request at commit 9fc1bbf4818a8f6ceea85ef6c57d0b8ed2f77160
is automatically being deployed to Dagster Cloud.
Location | Status | Link | Updated |
---|---|---|---|
demo_assets | ![]() |
View in Cloud | Jun 06, 2024 at 11:06 PM (UTC) |
basics | ![]() |
View in Cloud | Jun 06, 2024 at 11:06 PM (UTC) |
batch_enrichment | ![]() |
View in Cloud | Jun 06, 2024 at 11:06 PM (UTC) |
data-eng-pipeline | ![]() |
View in Cloud | Jun 06, 2024 at 11:06 PM (UTC) |
snowflake_insights | ![]() |
View in Cloud | Jun 06, 2024 at 11:06 PM (UTC) |
this is awesome!
One question -- is the goal of this PR that all of these assets (see below) have storage kinds? Also, I noticed that Sling doesn't -- I'm guessing that is a change to our sling integration and not something we can implement on our side though
Your pull request at commit b6e5aeacb51f35d8f725f86a2035aef56d052edd
is automatically being deployed to Dagster Cloud.
Location | Status | Link | Updated |
---|---|---|---|
demo_assets | ![]() |
View in Cloud | Jun 07, 2024 at 08:11 PM (UTC) |
basics | ![]() |
View in Cloud | Jun 07, 2024 at 08:11 PM (UTC) |
batch_enrichment | ![]() |
View in Cloud | Jun 07, 2024 at 08:11 PM (UTC) |
data-eng-pipeline | ![]() |
View in Cloud | Jun 07, 2024 at 08:11 PM (UTC) |
snowflake_insights | ![]() |
View in Cloud | Jun 07, 2024 at 08:11 PM (UTC) |
this is awesome!
One question -- is the goal of this PR that all of these assets (see below) have storage kinds? Also, I noticed that Sling doesn't -- I'm guessing that is a change to our sling integration and not something we can implement on our side though
![]()
I updated all assets (with a couple exceptions) to include storage_kind since the compute_kind doesn't necessarily have to match the storage_kind. That said, dagstermill_assets (model_nb) aren't yet supported and I'm still unsure how we want to represent the k8s storage_kind.
Your pull request at commit 7658c221a871f2cf951ccf5e3f258fc92a6ae3ec
is automatically being deployed to Dagster Cloud.
Location | Status | Link | Updated |
---|---|---|---|
demo_assets | ![]() |
View in Cloud | Jun 07, 2024 at 09:00 PM (UTC) |
basics | ![]() |
View in Cloud | Jun 07, 2024 at 09:00 PM (UTC) |
batch_enrichment | ![]() |
View in Cloud | Jun 07, 2024 at 09:00 PM (UTC) |
data-eng-pipeline | ![]() |
View in Cloud | Jun 07, 2024 at 09:00 PM (UTC) |
snowflake_insights | ![]() |
View in Cloud | Jun 07, 2024 at 09:00 PM (UTC) |
Adding
storage_kind
to non-dbt assets in thehooli_data_eng
code location.Summary of changes:
config_utils.py
to dynamically generate storage_kind (duckdb or snowflake) based on deployment (local or branch/prod)compute_kind
didn't make this obviousOne thing to note, I defaulted the storage_kind to S3 for assets using the
model_io_manager
since it uses theFilesystemIOManager()
locally and S3 in branch and prod deployments.