apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.88k stars 4.63k forks source link

[DSIP-56] Refactor JDBC registry support session timeout and data change event #16278

Closed ruanwenjun closed 4 months ago

ruanwenjun commented 4 months ago

Search before asking

Motivation

DS have support to jdbc registry which will rely on mysql/pg to accomplice the service discovery. But there exist some problem in current implementation.

Design Detail

image

JdbcRegistryServer

The JdbcRegistryServer represent the server side of JdbcRegistry, each service instance should create a JdbcRegistryServer. The JdbcRegistryServer is responsible to maintain the client's heart and response for the client's request.

Once the client's heartbeat doesn't refreshed success after client's session timeout, then the JdbcRegistryServer will clear the client's relation data.

JdbcRegistryClient

We can use JdbcRegistryClient to creat/delete data、acquire lock、 subscribe the data change event, connection change event.

t_ds_jdbc_registry_client_heartbeat

t_ds_jdbc_registry_client_heartbeat will store the active jdbc client heartbeat, once the jdbc client is dead , the server will delete the record in t_ds_jdbc_registry_client_heartbeat.

t_ds_jdbc_registry_data_change_event

t_ds_jdbc_registry_data_change_event will store the change event of t_ds_jdbc_registry_data, the server will detect the incremental events in this table and trigger data change event. Once we create/delete/update the record in t_ds_jdbc_registry_data, will insert a relation event to t_ds_jdbc_registry_data_change_event. Once the event is insert after 2h then it will be deleted.

Compatibility, Deprecation, and Migration Plan

Not compatibility with the old version, need to recreate the registry table.

Test Plan

No response

Code of Conduct