moiot / gravity

A Data Replication Center
Apache License 2.0
914 stars 172 forks source link

MySQL -> ES, 自定义的 ES output,执行时 Execute 函数接受到 Msgs 数量为1 #254

Open Stanley3 opened 4 years ago

Stanley3 commented 4 years ago

MySQL Input 的配置

[input]
    type = "mysql"
    mode = "batch"

[input.config]
    nr-scanner = 10
    table-scan-batch = 10000
    batch-per-second-limit = 10
    max-full-dump-count = 10000

ES Output 的配置

[output]
    type="custom-elasticsearch"

[output.config]
    ignore-bad-request = true

Scheduler 的配置

[scheduler]
    type = "batch-table-scheduler"

[scheduler.config]
    nr-worker = 10
    batch-size = 1000
    queue-size = 1024
    sliding-window-size = 10240
    nr-retries = 3
    retry-sleep = "2s"

按照这个配置,在执行到 func (output *customESOutput) Execute(msgs []*core.Msg) error 函数时,msgs 数量总是为1,导致 ES 的 bulk request 形同虚设。 请问,应该如何调整配置,解决这个问题。

Ryan-Git commented 4 years ago

贴下监控?配置没看出啥问题