search

dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.67k stars 1.47k forks source link

Document and/or improve ergonomics to define upstream deps/amps for individually mapped dlt assets #23760

Open cmpadden opened 2 months ago

cmpadden commented 2 months ago

I find it very troublesome to define upstream dependencies and auto materialization for individual mapped dlt asset.

In dbt, I can define auto materialization and dependencies in model schema.yml for each individual model. And I don't need to write a specific translator. But I don't find a way to define them for each dlt resource. If I define them in translator, it means same auto materialization and same dependencies for a batch of dlt resource.

Unless I write a dlt_asset for every single dlt resource, I can control auto materialization and dependencies on a per dlt resource grain. But then I don't see the point of using this integration library anymore, I can just write generic dagster asset defs.

Originally posted by @wj-c in https://github.com/dagster-io/dagster/discussions/17300#discussioncomment-10354456

sryza commented 1 month ago

Hey @wj-c – I'm curious, is an API you have in mind that that would be ideal for this?