search

databrickslabs / dlt-meta

This is metadata driven DLT based framework for bronze/silver pipelines
Other
125 stars 54 forks source link

Schema getting overwritten with new tables when running with different data flow group #32

Closed vijaykishorektk closed 5 months ago

vijaykishorektk commented 6 months ago

Hi Team, thank you so much for the UC branch merge. We are now able to work with UC without any issues or workarounds. However, when i tried running the pipeline for a different dataflow group, this new group overwrites whatever existed before it, instead of writing alongside the other tables. Could you let me know how to fix this?

ravi-databricks commented 5 months ago

@vijaykishorektk , Are you running same pipeline twice changing group parameter from one to another e.g. A1 to A2? During pipeline run DataflowPipeline does not write to Dataflowspec tables(metadata tables).

vijaykishorektk commented 5 months ago

@ravi-databricks , sorry for not providing sufficient context and for not phrasing my question more clearly. Let me try to explain with an example:

  1. I'm creating spec tables for both bronze and silver with 2 groups, A1 and A2 containing tables a,b and c,d respectively. So 4 records will be added to both the spec tables.
  2. When I run the DLT pipeline with group parameter as A1, tables a and b get created.
  3. When I run the DLT pipeline with group parameter as A2, tables c and d get created, however, tables a and b are dropped from the database.

I was wondering if this is intended behavior, if not, should I add some config parameter or do something else to not make this happen?

For more context, I'm using the newly merged UC compatible branch and using DLT with UC enabled.

ravi-databricks commented 5 months ago

@vijaykishorektk You need to launch two separate pipelines one with Group A1 and another with A2. You cannot run same pipeline changing group from A1 to A2. This would be same even if you use regular DLT and change targets from table a , b and rewrites to b, c. Please refer DAIS DEMO

vijaykishorektk commented 5 months ago

@ravi-databricks thank you for the clarification!