Open jeppe742 opened 3 weeks ago
Can you try it against at 0.18.0, if it still persist, then it deserves an upstream issue at delta-kernel-rs repo
@ion-elgreco It's also an issue with 0.18.0. Will try to create an issue in the delta-kernel-rs repo 😃
Environment
Delta-rs version: 0.17.4
Binding: Python
Environment:
Bug
What happened: We are investigating using Delta Uniform to have our Spark jobs also write Iceberg metadata. In order to enable the generation of Iceberg metadata you have to set the
delta.enableIcebergCompatV2
property on the table. When you set this, the Delta transaction log will include some more information.E.g if you run the Example from the Uniform documentation
You will get a delta transaction that looks something like this
If you try to read this table you get the following error
_internal.DeltaProtocolError: Invalid JSON in file stats: data did not match any variant of untagged enum MetadataValue at line 1 column 147
Seems like what is causing this is that Delta adds
"delta.columnMapping.nested.ids":{}
to the metaData config, but the delta kernel doesn't support nested structs in meta dataWhat you expected to happen: I should be able to read a delta table written with uniform enabled
How to reproduce it:
More details: