search

apache / celeborn

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.
https://celeborn.apache.org/
Apache License 2.0
862 stars 351 forks source link

[CELEBORN-1550][FOLLOWUP] Support celeborn.dynamicConfig.store.backend with short name and backend implementation #2698

Closed SteNicholas closed 3 weeks ago

SteNicholas commented 3 weeks ago

What changes were proposed in this pull request?

Introduce ConfigStore to support celeborn.dynamicConfig.store.backend with short name and backend implementation.

Why are the changes needed?

celeborn.dynamicConfig.store.backend is allowed to be specified in two ways:

Therefore, it's recommended to introduce ConfigStore based on SPI mechanism for celeborn.dynamicConfig.store.backend instead of dynamicConfigStoreBackendShortNames which could not add other short name of backend implementation for users.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

CI.

SteNicholas commented 3 weeks ago

Ping @RexXiong, @s0nskar.

s0nskar commented 3 weeks ago

@SteNicholas Really interesting PR, i was not aware about the java.util.ServiceLoader.

Can we update the config description and website docs to reflect how to use this functionality.

SteNicholas commented 3 weeks ago

@s0nskar, thanks for review. I have updated the description of celeborn.dynamicConfig.store.backend. Meanwhile, I would like to introduce ConfigStore document in another pull request. PTAL.

SteNicholas commented 3 weeks ago

Ping @RexXiong, @FMX.

RexXiong commented 3 weeks ago

Merge to main(v0.6.0) and branch-0.5(v0.5.2)