search

deephaven / deephaven-core

Deephaven Community Core
Other
257 stars 80 forks source link

iceberg primitive array support #5871

Open devinrsmith opened 3 months ago

devinrsmith commented 3 months ago

We are currently unable to deserialize byte[], and likely other primitive array types.

Value: io.deephaven.engine.table.impl.locations.TableDataException: Unsupported iceberg column type LIST
    at io.deephaven.iceberg.util.IcebergCatalogAdapter.convertPrimitiveType(IcebergCatalogAdapter.java:161)
    at io.deephaven.iceberg.util.IcebergCatalogAdapter.fromSchema(IcebergCatalogAdapter.java:108)
    at io.deephaven.iceberg.util.IcebergCatalogAdapter.readTableInternal(IcebergCatalogAdapter.java:506)
    at io.deephaven.iceberg.util.IcebergCatalogAdapter.readTable(IcebergCatalogAdapter.java:380)
    at io.deephaven.iceberg.util.IcebergCatalogAdapter.readTable(IcebergCatalogAdapter.java:394)

While I don't think we necessarily need nested array support, we should probably support at least single primitive arrays?

devinrsmith commented 3 months ago

We might consider that the catalog type (as written by pyiceberg) for a byte[] type would be better represented as a BinaryType (as opposed to List) in catalog.

That said, we should consider the same scenario but with short[].