Silver Transformations Became a Dependency

databrickslabs / dlt-meta

This is metadata driven DLT based framework for bronze/silver pipelines

Other

125 stars 54 forks source link

Silver Transformations Became a Dependency #42

Closed nathan-lammers closed 2 months ago

nathan-lammers commented 3 months ago

TypeError: __init__() missing 1 required positional argument: 'dataQualityExpectations',None,Map(),Map(),List(),List(),Map())

When a Delta Live Table is onboarded thru a previous version of DLT-META without silver data quality expectations, the code will fail when invoking the launch notebook.

ravi-databricks commented 2 months ago

One solution to this issue would be re-create onboarding tables since schema has changed. we added silver_data_quality_rules column to silver dataflowspec table.

nathan-lammers commented 2 months ago

Thanks Ravi!