search

apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://datastrato.ai/docs/
Apache License 2.0
681 stars 208 forks source link

[Improvement] shadow iceberg package in bundle catalog #4084

Open FANNG1 opened 3 days ago

FANNG1 commented 3 days ago

What would you like to be improved?

4005 will move some Iceberg common logic like IcebergCatalogBackend from catalog-lakehouse-iceberg toiceberg-common module, this cause spark iceberg IT failure in embedded mode for IcebergCatalogBackend is not found. The main reason is IcebergCatalogPropertiesMetadata is placed in bundle-catalog and is loaded by base classloader, so the IcebergCatalogBackend referenced by IcebergCatalogPropertiesMetadata will be loaded by base classloader which couldn't load IcebergCatalogBackend. To fix this we shadow iceberg package in bundle catalog.

How should we improve?

No response

FANNG1 commented 3 days ago

Why everything is ok before split iceberg common logic? because before split , IcebergCatalogBackend is packaged in bundle-catalog and could be loaded by base classloader. After split, IcebergCatalogBackend is no longer packaged and couldn't be found by base classloader.