search

trinitylake-io / trinitylake

Open LakeHouse Format for Big Data Analytics, ML & AI
https://trinitylake.io/
Apache License 2.0
4 stars 2 forks source link

Create Iceberg REST Catalog server #5

Open jackye1995 opened 5 days ago

jackye1995 commented 5 days ago

There is a pending decision for if we should use Java or Python for the server. I am currently leaning towards doing that in Python with FastAPI.

jackye1995 commented 5 days ago

But we will develop Java SDK first, so probably we will have to go with Java

jackye1995 commented 5 days ago

Python example: https://github.com/kevinjqliu/iceberg-rest-catalog

Java example: https://github.com/databricks/iceberg-rest-image/tree/master/src/main/java/org/apache/iceberg/rest

BsoBird commented 2 days ago

Question. Do we have to use rest catalog?Can't we use a module similar to hadoop_catalog in iceberg to access iceberg?

BsoBird commented 2 days ago

Because, in my opinion, it is possible to achieve reliable catalog management while relying only on the filesystem. If we expose the rest-catalog specification interface to the public, it may not be too attractive to users.We should mainly expose the fs-catalog specification to users, the rest-catalog specification is just an additional feature.