Handling historical silver transformations with Full Refresh

[WilliamMize]

I've built a process to generate our silver_transformations.json file. I have a question/future issue for what a DLT "Full refresh" could do if you have silver_transformations that might change for a table over time. If a "Full refresh" is done, then I believe it would only grab what is current in the dataflowspecTable for silver, which could miss any historical transformations put in place at their respective times. Am I missing something architecturally with DLT or DLT-meta that can handle this use case? I appreciate any help/conversation surrounding this topic.

[ravi-databricks]

@WilliamMize , yes! as part of full refresh it will apply latest silver transformations present in silver_dataflow_spec to full refresh. There are two way you can look into this

• if you have modified your silver transformation logic then you want your whole data to follow silver transformation.
• If you want to retain history then full refresh will mess up history as its reprocess. Same would be true for regular DLT pipeline too. e.g you have v1 dlt notebook having some transformation logic and now with v2 you have different logic.

[WilliamMize]

@ravi-databricks, I appreciate your input! I often find myself preemptively strategizing for potential issues in our data architecture. There are two approaches I'm considering to enhance our system's resilience and flexibility:

• Expanding the silver_dataflow_spec Configuration: If we modify the silver_dataflow_spec to accept multiple entries for a single table, could this provide the necessary flexibility for creating varied views feeding into the silver layer? This could potentially leverage a CDC timestamp to delineate distinct transformation segments.
• Creating a historical table for older Transformation: Another idea is to clone the existing table, preserving it as a backup outside the Delta Live Tables (DLT) environment. This allows us to implement extensive transformations on the 'new' version of the table, without risking the integrity of the original data. Post-transformation, we could merge insights from both the original and modified tables downstream.

Your thoughts on these approaches would be greatly appreciated.

[ravi-databricks]

@WilliamMize , There is version option while onboarding dataflowspec that means you can have multiple versions for same input-outputs. You need to increment versions so that new dataflowspec will be applied against pipeline

[WilliamMize]

@ravi-databricks Thank you for your help! I've got dlt-meta setup for our first system in production now.

[ravi-databricks]

Awesome!! @WilliamMize would love to hear about your journey to prod with dlt-meta.