dlt-hub / dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
https://dlthub.com/docs
Apache License 2.0
2.38k stars 154 forks source link

No schema evolution with `delta` table format on `filesystem` destination with `merge` write disposition #1739

Closed jorritsandbrink closed 1 month ago

jorritsandbrink commented 1 month ago

dlt version

0.5.4a0

Describe the problem

Schema evolution doesn't work for the delta table format on the filesystem destination when using the merge write disposition. New columns (that emerge after initial creation of the table) are disregarded—they don't end up in the table.

The append and replace dispositions don't have this issue.

Expected behavior

No response

Operating system

Windows

Runtime environment

Local

Python version

3.8

dlt data source

No response

dlt destination

Filesystem & buckets

Other deployment details

No response

Additional information

No response