Config memagement with Nacos

[yuzhichang]

Nacos golang client: https://github.com/nacos-group/nacos-sdk-go

[ArtemChekunov]

Maybe consul ?

[yuzhichang]

@ArtemChekunov Unfortunately consul is unavailable in our product.

[sundy-li]

We can abstract the interface of configs, like IConfig, HttpConfig, LocalConfig, NacosConfig

[yuzhichang]

This issue can be combined with #84 .

I suggest following Config structure which all clickhouse_sinker instances share(fetched from Nacos, Consul, ZK etc.):

type Config struct {
    Kafka      map[string]*KafkaConfig
    Clickhouse map[string]*ClickHouseConfig
    Tasks []*TaskConfig
    Common struct {
        FlushInterval     int
        BufferSize        int
        MinBufferSize     int
        MsgSizeHint       int
        ConcurrentParsers int
        LayoutDate        string
        LayoutDateTime    string
        LayoutDateTime64  string
        LogLevel          string
    }
    Assigns map[string][]string  //map instance_name to a list of task_name
}

Each instance can run multiple tasks. Each task can be assigned to multiple instances. Each task declares how many instances it needs.

The coordinator

• The coordinator (ckman, another project) provides API and/or webui to add/delete/modify tasks.
• The coordinator watches (do service discovery) instance startup/disappear events, and assign tasks to instances (do publish config).

The schedule platform

The schedule platform(or maybe manually, outside my concern) start some clickhouse_sinker instances and start another one if a instance fail.

clickhouse_sinker

• Every clickhouse_sinker instance register itself to a service manager (Nacos, Consul, ZK etc.).
• clickhouse_sinker watches (do get config) cofig, compare its assignment with the current one, apply changes.
• clickhouse_sinker shall try to recover a task in endless loop if the task fail. This ensures the tasks run on an instance match the coordinator assigned.

[yuzhichang]

Implemented with #86.