[Feature]: Support multi-instance AMS REST service

link3280 commented 5 months ago

Description

Currently, only the leader AMS would provide REST service. When an AMS leader transition happens, the standby AMS would take 30+ seconds to initialize optimizing, after which the REST service is available. That affects the availability of the REST service.

I proposed to make the REST service independent of the leadership. That's to say, the stand-by AMS instances need to initialize and run REST service on start. Their only difference to a leader instance is no optimizing service.

Use case/motivation

No response

Describe the solution

I drew a diagram of the main modules in AMS.

amoro-follower-rest (1)

The green ones are the modules that should be run by both the leader and the followers when starting.
The yellow ones are the leader-only services that should be run when an AMS becomes the leader.
The remaining red one is a new module that syncs catalog changes from the leader through reloading from the database.

Subtasks

No response

Related issues

No response

Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

baiyangtx commented 3 months ago

Are you still working on this? I'm interesting in this too. @link3280

link3280 commented 3 months ago

Are you still working on this? I'm interesting in this too. @link3280

I'm reconsidering the proposal since @majin1102 plans to support multi-instance AMS. We could align the efforts to avoid duplicate work. What do you think? @baiyangtx

baiyangtx commented 3 months ago

I started a discussion and provided a design. #2997

You are welcome to participate in the discussion. @link3280 @majin1102

czy006 commented 1 month ago

+1 We also need this part, design can be completed together will reduce duplication of work

apache / amoro