search

pingcap / tidb-operator

TiDB operator creates and manages TiDB clusters running in Kubernetes.
https://docs.pingcap.com/tidb-in-kubernetes/
Apache License 2.0
1.22k stars 493 forks source link

Pump deployment fails: silently failing due to write binlog error no available space #656

Closed sokada1221 closed 5 years ago

sokada1221 commented 5 years ago

Bug Report

What version of Kubernetes are you using?

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-19T22:12:47Z", GoVersion:"go1.12.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-eks-c57ff8", GitCommit:"c57ff8e35590932c652433fab07988da79265d5b", GitTreeState:"clean", BuildDate:"2019-06-07T20:43:03Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

What version of TiDB Operator are you using?

$ kubectl exec -n tidb-admin tidb-controller-manager-7b998f5747-fsktc -- tidb-controller-manager -V
TiDB Operator Version: version.Info{TiDBVersion:"2.1.0", GitVersion:"v1.0.0-beta.3", GitCommit:"6257dfaad68f55f745f20f6f5d19b10bea2b0bea", GitTreeState:"clean", BuildDate:"2019-06-06T09:51:04Z", GoVersion:"go1.12", Compiler:"gc", Platform:"linux/amd64"}

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

$ kubectl get sc
NAME            PROVISIONER                    AGE
ebs-gp2         kubernetes.io/aws-ebs          19h
gp2 (default)   kubernetes.io/aws-ebs          19h
local-storage   kubernetes.io/no-provisioner   19h
$ kubectl get pvc -n stg4tidb
NAME                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
data-stg4tidb-pump-0   Bound    pvc-57fb40ee-a5ef-11e9-9131-060d2ce056e4   10Gi       RWO            ebs-gp2         60m
data-stg4tidb-pump-1   Bound    pvc-6412c9d2-a5ef-11e9-9131-060d2ce056e4   10Gi       RWO            ebs-gp2         60m
pd-stg4tidb-pd-0       Bound    pvc-c2367f92-a55d-11e9-9131-060d2ce056e4   1Gi        RWO            ebs-gp2         18h
pd-stg4tidb-pd-1       Bound    pvc-c23b2325-a55d-11e9-9131-060d2ce056e4   1Gi        RWO            ebs-gp2         18h
pd-stg4tidb-pd-2       Bound    pvc-c23f4d3b-a55d-11e9-9131-060d2ce056e4   1Gi        RWO            ebs-gp2         18h
stg4tidb-monitor       Bound    pvc-c2069a71-a55d-11e9-8a1c-0a3a29ff5ab2   100Gi      RWO            ebs-gp2         18h
tikv-stg4tidb-tikv-0   Bound    local-pv-7ef562a2                          1741Gi     RWO            local-storage   18h
tikv-stg4tidb-tikv-1   Bound    local-pv-dfc17131                          1741Gi     RWO            local-storage   18h
tikv-stg4tidb-tikv-2   Bound    local-pv-a116ff25                          1741Gi     RWO            local-storage   18h

What's the status of the TiDB cluster pods?

$ kubectl get po -n stg4tidb -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   IP              NODE                                               NOMINATED NODE   READINESS GATES
stg4tidb-discovery-c4c55c845-sp4ls   1/1     Running   0          18h   10.220.18.55    ip-10-220-18-177.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-monitor-d6bfd6865-9hnh7     2/2     Running   0          58m   10.220.18.246   ip-10-220-18-177.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-pd-0                        1/1     Running   2          18h   10.220.19.87    ip-10-220-19-116.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-pd-1                        1/1     Running   0          18h   10.220.17.208   ip-10-220-17-112.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-pd-2                        1/1     Running   0          18h   10.220.18.101   ip-10-220-18-5.ap-northeast-1.compute.internal     <none>           <none>
stg4tidb-pump-0                      1/1     Running   0          80s   10.220.19.223   ip-10-220-19-150.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-pump-1                      1/1     Running   0          67s   10.220.18.54    ip-10-220-18-29.ap-northeast-1.compute.internal    <none>           <none>
stg4tidb-tidb-0                      1/1     Running   0          10m   10.220.18.45    ip-10-220-18-150.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-tidb-1                      1/1     Running   0          48s   10.220.19.239   ip-10-220-19-133.ap-northeast-1.compute.internal   <none>           <none>
stg4tidb-tikv-0                      1/1     Running   0          18h   10.220.17.190   ip-10-220-17-50.ap-northeast-1.compute.internal    <none>           <none>
stg4tidb-tikv-1                      1/1     Running   0          18h   10.220.18.36    ip-10-220-18-49.ap-northeast-1.compute.internal    <none>           <none>
stg4tidb-tikv-2                      1/1     Running   0          18h   10.220.19.29    ip-10-220-19-246.ap-northeast-1.compute.internal   <none>           <none>

What did you do?

  1. Deploy TiDB operator and TiDB cluster with pump enabled Relevant values used for installing TiDB cluster: 
    binlog:
pump:
create: false
replicas: 2
image: pingcap/tidb-binlog:v2.1.14
imagePullPolicy: IfNotPresent
logLevel: info
# storageClassName is a StorageClass provides a way for administrators to describe the "classes" of storage they offer.
# different classes might map to quality-of-service levels, or to backup policies,
# or to arbitrary policies determined by the cluster administrators.
# refer to https://kubernetes.io/docs/concepts/storage/storage-classes
storageClassName: ebs-gp2
storage: 10Gi
syncLog: true
# a integer value to control expiry date of the binlog data, indicates for how long (in days) the binlog data would be stored.
# must bigger than 0
gc: 7
# number of seconds between heartbeat ticks (in 2 seconds)
heartbeatInterval: 2
  2. Pump and TiDB pods are both running okay but TiDB becomes inaccessible due to pump

What did you expect to see? Successful deployment with correctly functioning pump

What did you see instead? Pump log is filled with these error messages:

runtime.main
    /usr/local/go/src/runtime/proc.go:188
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1337
2019/07/14 05:13:17 server.go:239: [error] write binlog error no available space
github.com/pingcap/tidb-binlog/pump/storage.init.ializers
    /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb-binlog/pump/storage/errors.go:10
runtime.main
    /usr/local/go/src/runtime/proc.go:188
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1337
2019/07/14 05:13:17 server.go:239: [error] write binlog error no available space
github.com/pingcap/tidb-binlog/pump/storage.init.ializers
    /home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb-binlog/pump/storage/errors.go:10
runtime.main
    /usr/local/go/src/runtime/proc.go:188
runtime.goexit
    /usr/local/go/src/runtime/asm_amd64.s:1337

Eventually, it starts printing the following info messages:

2019/07/14 05:13:21 storage.go:356: [info] DBStats: {WriteDelayCount:0 WriteDelayDuration:0s WritePaused:false AliveSnapshots:0 AliveIterators:0 IOWrite:38920 IORead:144245 BlockCacheSize:13003 OpenedTablesCount:1 LevelSizes:[36962] LevelTablesCounts:[1] LevelRead:[0] LevelWrite:[0] LevelDurations:[0s]}
2019/07/14 05:13:21 server.go:498: [info] writeBinlogCount: 26641, alivePullerCount: 0,  maxCommitTS: 409752353337508548
2019/07/14 05:13:31 storage.go:356: [info] DBStats: {WriteDelayCount:0 WriteDelayDuration:0s WritePaused:false AliveSnapshots:0 AliveIterators:0 IOWrite:39412 IORead:144245 BlockCacheSize:13003 OpenedTablesCount:1 LevelSizes:[36962] LevelTablesCounts:[1] LevelRead:[0] LevelWrite:[0] LevelDurations:[0s]}
2019/07/14 05:13:31 server.go:498: [info] writeBinlogCount: 26641, alivePullerCount: 0,  maxCommitTS: 409752360415133697

TiDB log is filled with these warning messages:

2019/07/14 05:27:30.499 client.go:288: [warning] [pumps client] write binlog to pump stg4tidb-pump-1:8250 (type: Prewrite, start ts: 409752581022679048, commit ts: 0, length: 63077) error rpc error: code = Unknown desc = no available space

Eventually TiDB stops responding as it fails to communicate with pump:

[2019/07/14 05:27:45.538 +00:00] [INFO] [2pc.go:635] ["2PC clean up done"] [conn=27] [txnStartTS=409752581704253441]
2019/07/14 05:27:45.542 client.go:321: [info] [pumps client] write binlog to avaliable pumps all failed, will try unavaliable pumps
[2019/07/14 05:27:45.547 +00:00] [ERROR] [binloginfo.go:113] ["write binlog failed"] [error="write binlog failed, the last error no avaliable pump to write binlog"] [errorVerbose="write binlog failed, the last error no avaliable pump to write binlog\ngithub.com/pingcap/tidb-tools/tidb-binlog/pump_client.(*PumpsClient).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/tidb-tools@v2.1.3-0.20190116051332-34c808eef588+incompatible/tidb-binlog/pump_client/client.go:328\ngithub.com/pingcap/tidb/sessionctx/binloginfo.(*BinlogInfo).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/sessionctx/binloginfo/binloginfo.go:111\ngithub.com/pingcap/tidb/store/tikv.(*twoPhaseCommitter).prewriteBinlog.func1\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:743\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"] [stack="github.com/pingcap/tidb/sessionctx/binloginfo.(*BinlogInfo).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/sessionctx/binloginfo/binloginfo.go:113\ngithub.com/pingcap/tidb/store/tikv.(*twoPhaseCommitter).prewriteBinlog.func1\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:743"]
[2019/07/14 05:27:45.547 +00:00] [WARN] [session.go:382] ["commit failed"] [conn=20] ["finished txn"="Txn{state=invalid}"] [error="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog"] [errorVerbose="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog\ngithub.com/pingcap/errors.AddStack\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/errors.go:174\ngithub.com/pingcap/parser/terror.(*Error).GenWithStackByArgs\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190703021824-628683061f71/terror/terror.go:233\ngithub.com/pingcap/tidb/sessionctx/binloginfo.(*BinlogInfo).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/sessionctx/binloginfo/binloginfo.go:127\ngithub.com/pingcap/tidb/store/tikv.(*twoPhaseCommitter).prewriteBinlog.func1\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:743\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"]
2019/07/14 05:27:45.548 client.go:321: [info] [pumps client] write binlog to avaliable pumps all failed, will try unavaliable pumps
2019/07/14 05:27:45.548 adapter.go:413: [warning] # Txn_start_ts: 409752581560074260
# User: root@10.220.19.146
# Conn_ID: 20
# Query_time: 13.053171456
# Prewrite_time: 0.028558435 Write_keys: 3000 Write_size: 156000 Prewrite_region: 3
# DB: paylite_payment_migration
# Is_internal: false
# Digest: cac5c7d1c1782e36215435bcbec6c1be82b46486325beb8e04895bef8a620c16
# Num_cop_tasks: 0
# Cop_proc_avg: 0 Cop_proc_p90: 0 Cop_proc_max: 0
# Cop_wait_avg: 0 Cop_wait_p90: 0 Cop_wait_max: 0
"insert ignore into  acquiring_migration_status (orderId,status,version) values   (  3024886118766886912,'PUBLISHING',1 ),( 3024886118766895104,'PUBLISHING',1 ),( 3024886118766903296,'PUBLISHING',1 ),( 3024886118766919680,'PUBLISHING',1 ),( 3024886118766919681,'PUBLISHING',1 ),( 3024886118766919682,'PUBLISHING',1 ),( 3024886118766927872,'PUBLISHING',1 ),( 3024886118766944256,'PUBLISHING',1 ),( 3024886118766944257,'PUBLISHING',1 ),( 3024886118766944258,'PUBLISHING',1 ),( 3024886118766952448,'PUBLISHING',1 ),( 3024886118766952449,'PUBLISHING',1 ),( 3024886118766960640,'PUBLISHING',1 ),( 3024886118766968832,'PUBLISHING',1 ),( 3024886118766977024,'PUBLISHING',1 ),( 3024886118766993408,'PUBLISHING',1 ),( 3024886118767001600,'PUBLISHING',1 ),( 3024886118767009792,'PUBLISHING',1 ),( 3024886118767042560,'PUBLISHING',1 ),( 3024886118767042561,'PUBLISHING',1 ),( 3024886118767058944,'PUBLISHING',1 ),( 3024886118767058945,'PUBLISHING',1 ),( 3024886118767067136,'PUBLISHING',1 ),( 3024886118767075328,'PUBLISHING',1 ),( 3024886118767075329,'PUBLISHING',1 ),( 3024886118767091712,'PUBLISHING',1 ),( 3024886118767099904,'PUBLISHING',1 ),( 3024886118767108096,'PUBLISHING',1 ),( 3024886118767116288,'PUBLISHING',1 ),( 3024886118767132672,'PUBLISHING',1 ),( 3024886118767140864,'PUBLISHING',1 ),( 3024886127356755968,'PUBLISHING',1 ),( 3024886127356764160,'PUBLISHING',1 ),( 3024886127356788736,'PUBLISHING',1 ),( 3024886127356788737,'PUBLISHING',1 ),( 3024886127356846080,'PUBLISHING',1 ),( 3024886127356854272,'PUBLISHING',1 ),( 3024886127356854273,'PUBLISHING',1 ),( 3024886127356862464,'PUBLISHING',1 ),( 3024886127356878848,'PUBLISHING',1 ),( 3024886127356895232,'PUBLISHING',1 ),( 3024886127356903424,'PUBLISHING',1 ),( 3024886127356911616,'PUBLISHING',1 ),( 3024886127356928000,'PUBLISHING',1 ),( 3024886127356928001,'PUBLISHING',1 ),( 3024886127356928002,'PUBLISHING',1 ),( 3024886127356960768,'PUBLISHING',1 ),( 3024886127356985344,'PUBLISHING',1 ),( 3024886127356993536,'PUBLISHING',1 ),( 3024886127357009920,'PUBLISHING',1 ),( 30248861273570"(len:39081);
[2019/07/14 05:27:45.548 +00:00] [WARN] [session.go:834] ["run statement error"] [conn=20] [schemaVersion=72] [error="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog"] [errorVerbose="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog\ngithub.com/pingcap/errors.AddStack\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/errors.go:174\ngithub.com/pingcap/parser/terror.(*Error).GenWithStackByArgs\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190703021824-628683061f71/terror/terror.go:233\ngithub.com/pingcap/tidb/sessionctx/binloginfo.(*BinlogInfo).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/sessionctx/binloginfo/binloginfo.go:127\ngithub.com/pingcap/tidb/store/tikv.(*twoPhaseCommitter).prewriteBinlog.func1\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:743\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"] [session="{\n  \"currDBName\": \"paylite_payment_migration\",\n  \"id\": 20,\n  \"lastInsertID\": 47452074,\n  \"status\": 2,\n  \"strictMode\": true,\n  \"user\": {\n    \"Username\": \"root\",\n    \"Hostname\": \"10.220.19.146\",\n    \"CurrentUser\": false,\n    \"AuthUsername\": \"\",\n    \"AuthHostname\": \"\"\n  }\n}"]
[2019/07/14 05:27:45.548 +00:00] [ERROR] [conn.go:512] ["critical error, stop the server listener"] [conn=20] [error="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog"] [errorVerbose="[global:3]critical error write binlog failed, the last error no avaliable pump to write binlog\ngithub.com/pingcap/errors.AddStack\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/errors@v0.11.1/errors.go:174\ngithub.com/pingcap/parser/terror.(*Error).GenWithStackByArgs\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/pkg/mod/github.com/pingcap/parser@v0.0.0-20190703021824-628683061f71/terror/terror.go:233\ngithub.com/pingcap/tidb/sessionctx/binloginfo.(*BinlogInfo).WriteBinlog\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/sessionctx/binloginfo/binloginfo.go:127\ngithub.com/pingcap/tidb/store/tikv.(*twoPhaseCommitter).prewriteBinlog.func1\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/store/tikv/2pc.go:743\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"] [stack="github.com/pingcap/tidb/server.(*clientConn).Run\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/conn.go:512\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/server.go:383"]
[2019/07/14 05:27:45.548 +00:00] [INFO] [server.go:362] ["close connection"] [conn=20]
[2019/07/14 05:28:45.453 +00:00] [ERROR] [server.go:317] ["listener stopped, waiting for manual kill."] [stack="github.com/pingcap/tidb/server.(*Server).Run\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/server/server.go:317\nmain.runServer\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/tidb-server/main.go:552\nmain.main\n\t/home/jenkins/workspace/release_tidb_2.1-ga/go/src/github.com/pingcap/tidb/tidb-server/main.go:164\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"]

But pump has 10GiB of space configured and allocated. Disk space availability is correctly reported on Grafana, and it can also be confirmed inside the pod:

$ kubectl exec -n stg4tidb -it stg4tidb-pump-0 -- /bin/sh
/ # ps -ef | grep pump
    1 root      0:01 /pump -L=info -advertise-addr=stg4tidb-pump-0.stg4tidb-pump:8250 -config=/etc/pump/pump.toml -log-file=
   29 root      0:00 grep pump
/ # cat /etc/pump/pump.toml
# pump Configuration.

# addr(i.e. 'host:port') to listen on for client traffic
addr = "0.0.0.0:8250"

# addr(i.e. 'host:port') to advertise to the public
advertise-addr = ""

# a integer value to control expiry date of the binlog data, indicates for how long (in days) the binlog data would be stored.
# must bigger than 0
gc = 7

# path to the data directory of pump's data
data-dir = "/data"

# number of seconds between heartbeat ticks (in 2 seconds)
heartbeat-interval = 2

# a comma separated list of PD endpoints
pd-urls = "http://stg4tidb-pd:2379"

#[security]
# Path of file that contains list of trusted SSL CAs for connection with cluster components.
# ssl-ca = "/path/to/ca.pem"
# Path of file that contains X509 certificate in PEM format for connection with cluster components.
# ssl-cert = "/path/to/drainer.pem"
# Path of file that contains X509 key in PEM format for connection with cluster components.
# ssl-key = "/path/to/drainer-key.pem"
#
[storage]
# Set to `true` (default) for best reliability, which prevents data loss when there is a power failure.
sync-log = true
#
# we suggest using the default config of the embedded LSM DB now, do not change it unless you know what you are doing
# [storage.kv]
# block-cache-capacity = 8388608
# block-restart-interval = 16
# block-size = 4096
# compaction-L0-trigger = 8
# compaction-table-size = 67108864
# compaction-total-size = 536870912
# compaction-total-size-multiplier = 8.0
# write-buffer = 67108864
# write-L0-pause-trigger = 24
# write-L0-slowdown-trigger = 17
/ # cd /data
/data # df -h .
Filesystem                Size      Used Available Use% Mounted on
/dev/nvme1n1              9.7G     36.2M      9.7G   0% /data
xiaojingchen commented 5 years ago

@shinnosuke-okada what version of the pump?

sokada1221 commented 5 years ago

@xiaojingchen 

    image: pingcap/tidb-binlog:v2.1.14
xiaojingchen commented 5 years ago

@shinnosuke-okada the new version pump config have a parameter stop-write-at-available-space,

# stop write when disk available space less then the configured size
# 42 MB -> 42000000, 42 mib -> 44040192
# default: 10 gib
# stop-write-at-available-space = "10 gib"

so if the disk space is less than 10G, the pump will stop writing. you need to use a larger disk or change the parameter. But current operator can not support to change this parameter directly.

xiaojingchen commented 5 years ago

In production environments, it is recommended to allocate larger disks for Pump; In the other hand, we will improve Operator's configuration modification to make it more flexible and easy to use. this is a relate PR: #638

sokada1221 commented 5 years ago

@xiaojingchen Thanks a lot. These are very helpful! Let me see if I can unblock myself based on your information.

sokada1221 commented 5 years ago

@xiaojingchen I worked around by allocating more disk space for pump services. I think we have 2 issues here:

  1. tidb-cluster's default pump config shouldn't come with 10GiB
  2. pump should print a better error message for the case when stop-write-at-available-space is causing the no available space error

I'll make a PR for issue 1 immediately, and consider that as a solution to this issue. Planning to increase default disk size for pumps from 10GiB to 20GiB.

xiaojingchen commented 5 years ago

@shinnosuke-okada thanks for your feedback, that is very useful to us, and please create an new issue if it's convenient for you.

sokada1221 commented 5 years ago

Logged https://github.com/pingcap/tidb-binlog/issues/679 to address the error message issue. As the problem on TiDB-operator side is addressed, closing this ticket.