search

databrickslabs / dlt-meta

This is metadata driven DLT based framework for bronze/silver pipelines
Other
125 stars 54 forks source link

Merge option for onboarding json into metadata table #16

Open Lackshu opened 9 months ago

Lackshu commented 9 months ago

Feature request, not bug. It would be good to be able to merge the onboarding Json into the metadata tables by key (data_flow_id). This way we don't have to overwrite the table and we could maintain central metadata tables for all source entities that we're ingesting. This could be done manually of course but would be nice to have it baked in into the framework.

ravi-databricks commented 8 months ago

dlt-meta takes approach of looking at target tables to check your flows , this is closely aligned to dlt implementation. You can fork and override merge function so that you can do it on Ids

walter-iseyon commented 4 months ago

I don't know if this could be related to this issue but what about having the onboarding file split into multiple files so that each new data source could be added independently? The onboarding process could ask if you have a single onboarding file or if you have a folder that contains multiple files that need to be merged into one large file. I find it could get unwieldy to manage one large onboarding file instead of multiple small files.