search

apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://gravitino.apache.org
Apache License 2.0
959 stars 302 forks source link

[#2730] Split Iceberg REST service #4005

Closed FANNG1 closed 2 months ago

FANNG1 commented 3 months ago

What changes were proposed in this pull request?

Split IcebergRESTService into one separate module, it could be managed with or without Gravitino server.

For user

  1. provide an isolated Iceberg REST server package or Gravitino package with directory iceberg-rest-server which contains Iceberg REST server. They both contain isolated config files and start&stop script.
  2. The configuration items start with gravitino.iceberg-rest. are treated as Iceberg REST server configs. like gravitino.iceberg-rest.catalog-backend, gravitino.iceberg-rest.httpPort, etc. Configurations for Iceberg REST server are same in isolated package and Gravitino package.

For developer

The core is RESTService which start JettyServer for Iceberg REST server,

Why are the changes needed?

Iceberg rest service is managed as an auxiliary service in Gravitino server , for the users who want to use Iceberg REST service only, it introduces an extra burden.

Fix: #2730

Does this PR introduce any user-facing change?

yes, will add document in https://github.com/apache/gravitino/pull/4113

How was this patch tested?

  1. existing tests
  2. test in local env as a Iceberg REST server

related docs

https://docs.google.com/document/d/1lyJwMaaJKfMqtnH9c7LwvnOHRKm7gh8Al4Sw3T1DFjM/edit

lmccay commented 3 months ago

@FANNG1 - I am interested in this change and would like to read some background on it. Is there a dev@g.a.o thread to DISCUSS such a large change? A design doc or proposal in wiki? Can you update the PR template with some details of what this is doing and why?

FANNG1 commented 3 months ago

@FANNG1 - I am interested in this change and would like to read some background on it. Is there a dev@g.a.o thread to DISCUSS such a large change? A design doc or proposal in wiki? Can you update the PR template with some details of what this is doing and why?

Hi, @lmccay , I have updated the doc in the PR template, you could refer to it.

FANNG1 commented 2 months ago

@jerryshao @caican00 please help to review when you have time, thanks.

FANNG1 commented 2 months ago

update Iceberg REST service document in https://github.com/apache/gravitino/pull/4113

jerryshao commented 2 months ago

I will take a try locally to take a deep look.

FANNG1 commented 2 months ago

@jerryshao @yuqi1129 , all comments are addressed, please help to review again

  1. iceberg-rest-server and iceberg-common are placed in iceberg module
  2. when package Gravitino server, iceberg-rest-server is placed in the package, the corresponding bin and conf files are placed in bin and conf directly.
  3. ./bin/gravition.sh start to start GravitinoServer with IcebergRESTService if corresponding config is enabled, ./bin/iceberg-rest-server start to start IcebergRESTServer only.
  4. when assemble IcebergRESTServer, licence readme are placed in package
  5. when assemble distribution, IcebergRESTServer also assembled.
jerryshao commented 2 months ago

@FANNG1 can you please rebase the code.

FANNG1 commented 2 months ago

@FANNG1 can you please rebase the code.

done

FANNG1 commented 2 months ago

@jerryshao , all comments are addressed, please help to review again.

FANNG1 commented 2 months ago

@jerryshao , all comments are addressed, please help to review again