moonlibs / config

Package for loading external lua config
10 stars 12 forks source link

Introduce API to retrieve vshard-topology for storage and router for etcd.cluster.vshard #30

Open ochaton opened 8 months ago

ochaton commented 8 months ago

We have several installations with moonlibs/config as Configuration Driver for multiple vshard-clusters. All of them have to copy-paste function build_sharding which retrieves entire configuration under etcd/prefix and constructs vshard acceptable configuration.

After introducing master_selection_policy = etcd.cluster.vshard and having config.etcd.get_all actually we are able to standardize the way every installation should configure it's vshard cluster.

I propose to specify configuration of vshard like that:

common:
  vshard: // main vshard section shared across all instances
    bucket_count: XXX // 
    collect_bucket_garbage_interval: ...
    collect_lua_garbage: ...
    sync_timeout: ...
    rebalancer_disbalance_threshold: ...
    rebalancer_max_sending: ...
    discovery_mode: ...
    consts: // the way to override vshard consts. The moonlibs/config should not override somehow vshard.consts by himself.
        BUCKET_CHUNK_SIZE: 3000
    weights: // the way to specify distances between zones (the higher the longer)
      region-1:
        region-2: 100 // distances between zones, (default is none)
clusters:
  shard_001:
    vshard:
      weight: 1 // weight of replicaset in terms of rebalancing/resharding (default none)
instances:
  instance_001_01:
    vshard:
      zone: "region-1" // replica weight in terms of router API (not the same as replicaset weight)

moonlibs/config should not call vshard.storage.cfg or vshard.router.cfg. It should provide API as config.vshard_get("storage") and config.vshard_get("router") to fetch constructed topology for storage or router instead.

ochaton commented 8 months ago

Maybe, it is better to name it config:sharding() as it is done in Tarantool 3.0 config