represent software-defined assets that are views

[sryza]

Unlike non-view software-defined assets, software-defined assets that are views don't need to become stale when their upstream assets are rematerialized – it's assumed that they automatically incorporate upstream data changes immediately, without needing to be materialized themselves.

Product implications of first-class support for view assets

• When an asset that's upstream of a view asset is re-materialized, the view asset shouldn't show the "Upstream data" label
• When an asset that's upstream of a view asset is re-materialized, the view asset shouldn't be auto-materialized

Possible API:

@asset(is_view=True)
def asset1():
    ...

What we've heard:

• https://elementl-workspace.slack.com/archives/C03CA4TVCAW/p1680282319689969
• siim.sillaots - https://dagster.slack.com/archives/C01U954MEER/p1684345122344499
  • Hey! I’m gonna describe a pipeline with three assets. Dagster asset that’s upstream of a view(dbt or dagster asset) that’s upstream of a dagster or dbt asset. In this scenario first asset is partitioned by 15min and third asset is a daily partition. I’m trying to figure out how to make it work. Can I define the view in dagster as an asset or is it something that can be done with dbt.
  • What I’m aiming for is that whenever first asset finishes a new partition then the third partition would be started by asset reconciliation sensor. If I add the view dagster asset to the reconc sensor then the pipeline “works” but the view will materialise every time upstream is updated. That isn’t a good solution as upstream gets updated every 15mins.
  • Perhaps someone has had experience with dagster/dbt and views

[ThomasRolfsnes-EP]

+1 to this! This would greatly simplify dbt runs in dagster (assuming dbt-views automatically get is_view=True)

[stufan]

+1

[ssillaots-boku]

Hey @sryza! Any updates if this will be a focus point in near future?

[geoHeil]

I would also love this feature! Especially for SQL/DBT usecases which are important for enterprises

[sryza]

@ssillaots-boku alas we don't have an ETA on this one yet

[slopp]

We should also consider changing how freshness policies are calculated for views. The "overdue" alert should only be fired in cases where the view's DDL has changed and the asset has not been materialized, OR the upstream has not been materialized. If the upstream is up to date and the code version is also up to date, the view is "fresh" regardless of the last time it was materialized.

Plus one from another Dagster Cloud enterprise customer for this issue

[geoHeil]

@slopp https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/hooli_data_eng/assets/dbt_assets.py#L114-L118 I like your approach here.

but: https://github.com/dagster-io/hooli-data-eng-pipelines/blob/master/hooli_data_eng/assets/dbt_assets.py#L158 is tedious. Is there any chance targeting views instead of tables could be more native than managing a ton of tags? Shouldn`t this be part of the manifest.json somewhere?

[mjclarke94]

There are some interesting questions on how staleness is propagated in a view which will be accessed in a partitioned manner. In some cases, it would be possible to simply propagate staleness from upstream assets, but I can also think of situations where there would be a need to separately calculate a DataVersion for each partition.

Could some of the machinery from ObservableSourceAssets be leveraged here?

[ssillaots-boku]

Hey, @sryza ! Bugging you again. Still not in the plans?

[the4thamigo-uk]

I would dearly love a solution to this one also