search

databrickslabs / dlt-meta

Metadata driven Databricks Delta Live Tables framework for bronze/silver pipelines
https://databrickslabs.github.io/dlt-meta/
Other
156 stars 71 forks source link

It appears that schema ddl is being required when using cdc_apply_changes #90

Closed mweirath closed 3 months ago

mweirath commented 3 months ago

I am trying to use cdc_apply_changes with a bronze table that we are using schema evolution (so no defined schema). Having a schema isn't required by Databricks for this operation, but it appears that the code is checking for a schema ddl before you can run use cdc_apply_changes.

ravi-databricks commented 3 months ago

@mweirath see if this fix resolves it! If you are using autoloader you dont need to provide schema ddl and apply_changes would work for evolving schema.

mweirath commented 3 months ago

Thanks I will work on getting this version pulled in and tested tomorrow.

mweirath commented 3 months ago

@ravi-databricks - I confirmed with my team that we were able to pull in the change, and everything appears to be working now. Thank you for the quick fix. I will keep an eye out for v0.9 to come out.

I'm not sure if you close these or me.

ravi-databricks commented 3 months ago

@mweirath you can close it with comments! thanks!

mweirath commented 3 months ago

Confirmed fixed, thank you.