search

duckdb / duckdb_delta

DuckDB extension for Delta Lake
MIT License
139 stars 15 forks source link

duckdb delta 1.1 Found unmasked nulls #84

Open djouallah opened 2 months ago

djouallah commented 2 months ago

what is this ? i am getting the same error when using duckdb 1.0.0, does the extension for duckdb 1 get upgraded too ?

IOException: IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table: '/lakehouse/default/Tables/dbo/result/'): Hit error: 2 (ArrowError) with message (Invalid argument error: Found unmasked nulls for non-nullable StructArray field "predicate")

djouallah commented 2 months ago

as a workaround, make sure to compact your delta table and duckdb should be able to read it

from deltalake import DeltaTable
dt = DeltaTable('xxxxxxxxxxxxxxxxxx',storage_options={"allow_unsafe_rename":"true"})
if len(dt.file_uris()) >= 50 :
            dt.optimize.compact()
            dt.vacuum()
            dt.cleanup_metadata()
            dt.create_checkpoint()
29antonioac commented 1 week ago

Commenting to confirm the issue still exists on 1.1.3 and extension version f71402e. For me it happens sometimes depending on the filter: on the partition column is okay but in other columns make the query to fail.

In my case I'm reading from GCP, I'm not able to reproduce in local so I can't provide a minimal example 😢