search

databrickslabs / dlt-meta

Metadata driven Databricks Delta Live Tables framework for bronze/silver pipelines
https://databrickslabs.github.io/dlt-meta/
Other
156 stars 71 forks source link

How to chain multiple silver tables after bronze table? #66

Closed ewang-loyalty closed 3 months ago

ewang-loyalty commented 4 months ago

Hi, is it possible to have more than one silver tables after a bronze table? For example, after customers_cdc, can I have customers silver table reading from customers_cdc and another customers_clean silver table reading from customers? If so, how do I define these in onboarding.json?

ravi-databricks commented 4 months ago

@ewang-loyalty, it is possible, please check below details:

Please take a look at this Unit Test.

You can run onboarding for additional silver customer_clean table by having onboarding file and silver transformation with filter condition for fan out.

Run onboarding for slilver layer in append mode so it will append to existing silver tables. When you launch DLT pipeline it will read silver onboarding and run DLT for bronze source and silver as target

ewang-loyalty commented 3 months ago

Thanks for creating the unit test example! Was able to clarify that this needs to be a separate onboarding.json during the call. Closing the issue as resolved.