search

duckdb / duckdb_delta

DuckDB extension for Delta Lake
MIT License
121 stars 14 forks source link

duckdb delta 1.1 Found unmasked nulls #84

Open djouallah opened 2 weeks ago

djouallah commented 2 weeks ago

what is this ? i am getting the same error when using duckdb 1.0.0, does the extension for duckdb 1 get upgraded too ?

IOException: IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table: '/lakehouse/default/Tables/dbo/result/'): Hit error: 2 (ArrowError) with message (Invalid argument error: Found unmasked nulls for non-nullable StructArray field "predicate")

djouallah commented 1 week ago

as a workaround, make sure to compact your delta table and duckdb should be able to read it

from deltalake import DeltaTable
dt = DeltaTable('xxxxxxxxxxxxxxxxxx',storage_options={"allow_unsafe_rename":"true"})
if len(dt.file_uris()) >= 50 :
            dt.optimize.compact()
            dt.vacuum()
            dt.cleanup_metadata()
            dt.create_checkpoint()