The position where the binlog replication starts when task-mode is incremental and the downstream database checkpoint does not exist. If the checkpoint exists, the checkpoint is used. If neither the meta configuration item nor the downstream database checkpoint exists, the migration starts from the latest binlog position of the upstream.
During the actual test, synchronization can occasionally be started normally. Occasionally, dm-master reports an error and then the cluster restarts. During the restart, the cluster crashes for more than ten seconds and dmctl reports an error.
» start-task roger_test_240227.yaml
fail to run: [start-task roger_test_240227.yaml]
Error: [code=48001:class=dmctl:scope=internal:level=high], Message: can't connect to x.x.x.84:8466, RawCause: context deadline exceeded, Workaround: Please check your network connection.
» query-status --more
can not query task's status(in sources [])
fail to run: [query-status --more]
Error: rpc error: code = Canceled desc = grpc: the client connection is closing
At present, I personally suspect that the difference is caused by inconsistent mariadb user permissions.
The mariadb permission is all and synchronization can be started normally.
GRANT ALL PRIVILEGES ON *.* TO 'dmsyncnew'@'%' IDENTIFIED BY PASSWORD ''
Only some permissions will cause the cluster to crash
GRANT SELECT, RELOAD, PROCESS, SHOW DATABASES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, SHOW VIEW ON *.* TO 'dmsyncnew'@'%' IDENTIFIED BY PASSWORD ''
The following is the dm-master log during the two tests
As stated in the documentation, incremental mode can start synchronization normally without setting binlog-name, binlog-pos, and binlog-gtid.
Even if parameters must be specified, an error message should be reported in dmctl instead of causing a crash.
What did you see instead?
No response
Versions of the cluster
DM version (run dmctl -V or dm-worker -V or dm-master -V):
dm version
v6.5.3
---
dm-master
Release Version: v6.5.3-20231229-5120c7b
Git Commit Hash: 5120c7bf52d74d30860f9fec917a4561abaddbb8
Git Branch: heads/refs/tags/v6.5.3-20231229-5120c7b
UTC Build Time: 2023-12-29 10:30:27
Go Version: go version go1.19.13 linux/amd64
Failpoint Build: false
---
dm-worker
Release Version: v6.5.3-20231229-5120c7b
Git Commit Hash: 5120c7bf52d74d30860f9fec917a4561abaddbb8
Git Branch: heads/refs/tags/v6.5.3-20231229-5120c7b
UTC Build Time: 2023-12-29 10:30:27
Go Version: go version go1.19.13 linux/amd64
Failpoint Build: false
---
dmctl
Release Version: v6.5.3-20230926-7e90ede
Git Commit Hash: 7e90ede585d3bb1cd152c8eeff88957839f16bd4
Git Branch: heads/refs/tags/v6.5.3-20230926-7e90ede
UTC Build Time: 2023-09-26 05:21:14
Go Version: go version go1.19.12 linux/amd64
Failpoint Build: false
Upstream MySQL/MariaDB server version:
--------------
mysql Ver 15.1 Distrib 5.5.68-MariaDB, for Linux (x86_64) using readline 5.1
Connection id:
Current database:
Current user:
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server: MariaDB
Server version: 10.1.9-MariaDBV1.0R012D003-20180427-1600 Source distribution
Protocol version: 10
Connection: via TCP/IP
Server characterset: utf8
Db characterset: utf8
Client characterset: utf8
Conn. characterset: utf8
TCP port: 3306
Uptime: 208 days 1 hour 49 min 1 sec
Threads: 2011 Questions: 20601214022 Slow queries: 1276853 Opens: 51871031 Flush tables: 1 Open tables: 10240 Queries per second avg: 1145.928
--------------
Downstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):
fubinzh
commented
9 months ago
What did you do?
The relevant description of incremental in DM official documentation can automatically start synchronization without setting binlog-name, binlog-pos, and binlog-gtid. https://docs.pingcap.com/tidb/v6.5/task-configuration-file-full
During the actual test, synchronization can occasionally be started normally. Occasionally, dm-master reports an error and then the cluster restarts. During the restart, the cluster crashes for more than ten seconds and dmctl reports an error.
At present, I personally suspect that the difference is caused by inconsistent mariadb user permissions. The mariadb permission is all and synchronization can be started normally.
Only some permissions will cause the cluster to crash
The following is the dm-master log during the two tests
What did you expect to see?
As stated in the documentation, incremental mode can start synchronization normally without setting binlog-name, binlog-pos, and binlog-gtid. Even if parameters must be specified, an error message should be reported in dmctl instead of causing a crash.
What did you see instead?
No response
Versions of the cluster
DM version (run
dmctl -Vordm-worker -Vordm-master -V):Upstream MySQL/MariaDB server version:
Downstream TiDB cluster version (execute
SELECT tidb_version();in a MySQL client):How did you deploy DM: tiup or manually?
Other interesting information (system version, hardware config, etc):
current status of DM cluster (execute
query-status <task-name>in dmctl)