search

open-metadata / openmetadata-spark-agent

Apache License 2.0
2 stars 2 forks source link

Support iceberg catalog of type Glue #15

Closed efthymiosh closed 2 months ago

efthymiosh commented 2 months ago

The agent does not support glue-backed iceberg catalogs, failing with:

24/04/23 12:17:37 ERROR PlanUtils3: Catalog glue is unsupported
io.openlineage.spark3.agent.lifecycle.plan.catalog.UnsupportedCatalogException: glue
    at io.openlineage.spark3.agent.lifecycle.plan.catalog.IcebergHandler.getDatasetIdentifier(IcebergHandler.java:83)
    at io.openlineage.spark3.agent.lifecycle.plan.catalog.CatalogUtils3.lambda$getDatasetIdentifier$2(CatalogUtils3.java:61)
    at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361)

where glue is a catalog of type glue.

This is supported in openlineage-spark starting from 1.8.0

ulixius9 commented 2 months ago

closing this ticket by https://github.com/open-metadata/openmetadata-spark-agent/pull/16