search

databrickslabs / dlt-meta

Metadata driven Databricks Delta Live Tables framework for bronze/silver pipelines
https://databrickslabs.github.io/dlt-meta/
Other
148 stars 66 forks source link

expect_or_quarantine #1

Closed thbeh closed 1 year ago

thbeh commented 1 year ago

Hi, not an error but I was looking at this and noticed you defined

_"expect_or_quarantine":{ "validoperation":"operation IN ('APPEND', 'DELETE', 'UPDATE')" } in one of the dqe. But when I tried, it throws

_AttributeError: module 'dlt' has no attribute 'expect_orquarantine',Map(),Map(),List(),List(),Map())

Is there something I missed?

ravi-databricks commented 1 year ago

are you using expect_or_quarantine using dlt API or using dlt-meta framework? There is no in-built api provided by dlt, DLT-META uses this json rule to produce separate table using dlt.expect_all_or_drop(expect_or_quarantine_dict) https://github.com/databricks/dlt-meta/blob/main/src/dataflow_pipeline.py#LL204C37-L204C37

thbeh commented 1 year ago

Oh...no worries. I thought it was a DLT feature. Nice work for dlt-meta.