tabular-io / iceberg-rest-image

Simple project to expose a catalog over REST using a Java catalog backend
Apache License 2.0
86 stars 38 forks source link

Can we manage multiple catalogs ? #40

Open quydx opened 11 months ago

quydx commented 11 months ago

When using Jdbc Catalog, I can have many catalogs as I want. Each catalog contains its own namespaces and tables.

image

But with REST catalog, I wonder if I can do the same thing like JDBC Catalog. I can't find any API to create catalog and show all catalogs. Can someone give me an ideas? Thank!

dupen01 commented 8 months ago

To achieve this goal, here's what I did: I changed the name of the rest catalog to be obtained from the environment variable, and then built the rest server as a docker image. If I need to manage multiple different catalogs, I can start multiple rest servers containers using different environment variables, and then my client can connect to anyone of them, thereby achieving the purpose of managing multiple catalogs. I hope this will help you.

...
private static final String ENV_REST_CATALOG_NAME = "REST_CATALOG_NAME";
String catalogName = System.getenv().get(ENV_REST_CATALOG_NAME);
        LOG.info("REST Catalog Name: {}", catalogName);
        return CatalogUtil.buildIcebergCatalog(catalogName, catalogProperties, new Configuration());
docker run -id \
-p 8181:8181 \
--name ice-rest \
--net lakenet \
--ip 192.168.10.16 \
-e REST_CATALOG_NAME=rest \
-e CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO \
-e CATALOG_WAREHOUSE=s3://lakehouse/iceberg/warehouse \
-e CATALOG_S3_ENDPOINT=http://192.168.10.12:9000 \
-e CATALOG_URI=jdbc:mysql://192.168.10.11:3306/iceberg \
...