search

apache / amoro

Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats.
https://amoro.apache.org/
Apache License 2.0
874 stars 290 forks source link

[Improvement]: Integrate with Nessie/Polaris REST catalog #3089

Open nqvuong1998 opened 3 months ago

nqvuong1998 commented 3 months ago

Search before asking

What would you like to be improved?

How should we improve?

No response

Are you willing to submit PR?

Subtasks

No response

Code of Conduct

nqvuong1998 commented 3 months ago

cc @zhoujinsong

czy006 commented 3 months ago

We may wait for Nessie/Polaris to enter the first version of the apache Incubator and have a stable api

zhoujinsong commented 3 months ago

Amoro already supported the custom catalog implementation for Iceberg Format. So you can choose Iceberg Format and custom metastore type for both Nessie and Polaris.

Here is an example of using RESTCatalog for Polaris:

image

And you can also set the catalog-impl to org.apache.iceberg.nessie.NessieCatalog

Of course, if this requirement is generic enough, we can also add separate "type" options for Nessie and Paloris (Rest).

nqvuong1998 commented 3 months ago

Hi @zhoujinsong

  1. Does Amoro support advanced authentication properties for the Nessie/Polaris REST catalog, such as Bearer authentication settings and OAuth2 settings?
  1. Does Amoro require a custom jar to run with Nessie, such as org.projectnessie.nessie-integrations:nessie-spark-extensions-3.5_2.12?