search

apache / iceberg-go

Apache Iceberg - Go
https://iceberg.apache.org/
Apache License 2.0
142 stars 34 forks source link

Iceberg Extensions #183

Open dwilson1988 opened 4 weeks ago

dwilson1988 commented 4 weeks ago

Feature Request / Improvement

There are some various table formats that extend Iceberg by allowing additional metadata to be added to various components, for example, Havasu. I (and I'm sure others) would like the ability to to extend metadata such as the table metadata, manifest lists, and manifest entries to read additional metadata. I can't speak for all iceberg extensions, but havasu extends iceberg by adding some additional metadata fields in:

  1. Table Metadata: https://github.com/wherobots/havasu/blob/main/spec.md#table-metadata
  2. Manifests: https://github.com/wherobots/havasu/blob/main/spec.md#manifests

The addition of the geometry bounds on the manifests is primarily for scan planning. It seems like at this point, until Iceberg support for geospatial types is added (which seems like it could be in the fairly distant future), augmenting scan planning may be a bit far away, but I'd propose that allowing metatadata extensions to table metadata and manifests maybe be worth consideration.

There are probably many ways this could be implemented, and I'm happy to take a stab at it since it's something I personally need on my project. First I wanted to ask if this is within the scope of this project or if you have a recommended path to something like this?

zeroshade commented 4 weeks ago

I'd have to think about it a bit as far as recommending a path. But I would definitely see this as within the scope of this implementation if we can come up with a good/useful way to do so.