search

apache / iceberg-rust

Apache Iceberg
https://rust.iceberg.apache.org/
Apache License 2.0
474 stars 97 forks source link

enhancement: caching of partition_schemas in `fn plan_files()` #361

Closed marvinlanhenke closed 2 months ago

marvinlanhenke commented 2 months ago

As of right now, the partition_schema is built in every iteration for each ManifestFile regardless if it has the same partition_spec.

This could be optimized as well by caching the partition_schema, similar to the partition filter.

Also the unnecessary clones (here, here) could be removed, or only be "lazily cloned" if we have a cache miss.