search

icelake-io / icelake

Pure Rust Iceberg Implementation
Apache License 2.0
166 stars 19 forks source link

Supports rest catalog. #120

Closed liurenjie1024 closed 1 year ago

liurenjie1024 commented 1 year ago

The current implementation only supports filesystem catalog. In production, jdbc/hive metastore catalogs are widely used. To support them, we need to support rest catalog so that we don't need to interact with them directly.

liurenjie1024 commented 1 year ago

The open api spec is here: https://github.com/apache/iceberg/blob/master/open-api/rest-catalog-open-api.yaml

We can use https://openapi-generator.tech/docs/installation#homebrew to generate rust code for it. But one thing to discuss is that whether we should keep the generated code in repo?

Personally I prefer to keep them in repo to simplify whole thing, how do you guys think?

cc @ZENOTME @Xuanwo

Xuanwo commented 1 year ago

Personally I prefer to keep them in repo to simplify whole thing, how do you guys think?

I perfer to keep them so that our users don't need to generate again.

ZENOTME commented 1 year ago

+1. Sounds reasonable for me.

liurenjie1024 commented 1 year ago

Cool, let's keep them

nooberfsh commented 1 year ago

Hi, Our team are experimenting Trino + Iceberg with Nessie Catalog (can be replaced with rest catalog), the query experience is good. The missing part is data ingestion, we want to use icelake to ingest data to our lakehouse, the missing rest catalog support prevent us from doing so. So what's the status about this? BTW: icelake is a great project.

liurenjie1024 commented 1 year ago

We have plans to support this, due to the schedule, maybe one month later we will start to implement it.

liurenjie1024 commented 1 year ago

Tracked in #172