Closes #3173. Removes the overwrite_schema kwarg from the Delta Table connector and bumps the deltalake version. Per this PR in the deltalake repo, the overwrite_schema kwarg is deprecated as of version 0.18.0. Users can specify schema_mode="merge" to obtain the same behavior.
schema_mode="merge" is equivalent to overwrite_schema=False
schema_mode="overwrite" is equivalent to overwrite_schema=True
Also adds an engine parameter that you can use to set "rust" or "pyarrow" as the engine. engine defaults to "pyarrow" and schema_mode defaults to None, which is consistent with the behavior in deltalake documented here.
Testing
The Delta Table ingest tests should pass on this PR.
Summary
Closes #3173. Removes the
overwrite_schemakwarg from the Delta Table connector and bumps thedeltalakeversion. Per this PR in thedeltalakerepo, theoverwrite_schemakwarg is deprecated as of version0.18.0. Users can specifyschema_mode="merge"to obtain the same behavior.schema_mode="merge"is equivalent tooverwrite_schema=Falseschema_mode="overwrite"is equivalent tooverwrite_schema=TrueAlso adds an
engineparameter that you can use to set"rust"or"pyarrow"as the engine.enginedefaults to"pyarrow"andschema_modedefaults toNone, which is consistent with the behavior indeltalakedocumented here.Testing
The Delta Table ingest tests should pass on this PR.