search

pingcap / dumpling

Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).
Apache License 2.0
280 stars 85 forks source link

使用dumpling 备份100g数据，经常出现 invalid connection的问题 #345

Closed hu177 closed 3 years ago

hu177 commented 3 years ago

每次dump表中的数据大小都超过100G，使用10次会报出2-3次的invalid connection错误， 使用命令如下： dumpling --filetype csv \ --tidb-mem-quota-query 838860800 \ -t 1 \ -r 2000000 \ -F 256MiB \ --sql 'select * from database.table`' \ --params "tidb_distsql_scan_concurrency=3"

详细日志如下： [2021/08/31 15:26:06.335 +08:00] [WARN] [writer_util.go:325] ["fail to dumping table(chunk), will revert some metrics now"] [error="invalid connection"] [errorVerbose="invalid connection\ngithub.com/pingcap/errors.AddStack\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/root/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/juju_adaptor.go:15\ngithub.com/pingcap/dumpling/v4/export.(*rowIter).Error\n\t/root/dumpling/v4/export/ir_impl.go:42\ngithub.com/pingcap/dumpling/v4/export.WriteInsertInCsv\n\t/root/dumpling/v4/export/writer_util.go:409\ngithub.com/pingcap/dumpling/v4/export.FileFormat.WriteInsert\n\t/root/dumpling/v4/export/writer_util.go:643\ngithub.com/pingcap/dumpling/v4/export.(*Writer).tryToWriteTableData\n\t/root/dumpling/v4/export/writer.go:217\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData.func1\n\t/root/dumpling/v4/export/writer.go:200\ngithub.com/pingcap/br/pkg/utils.WithRetry\n\t/root/go/pkg/mod/github.com/hu177/br@v1.3.3/pkg/utils/retry.go:46\ngithub.com/pingcap/dumpling/v4/export.(*Writer).WriteTableData\n\t/root/dumpling/v4/export/writer.go:161\ngithub.com/pingcap/dumpling/v4/export.(*Writer).handleTask\n\t/root/dumpling/v4/export/writer.go:103\ngithub.com/pingcap/dumpling/v4/export.(*Writer).run\n\t/root/dumpling/v4/export/writer.go:85\ngithub.com/pingcap/dumpling/v4/export.(*Dumper).startWriters.func4\n\t/root/dumpling/v4/export/dump.go:282\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/root/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"] [database=] [table=] ["finished rows"=917767] ["finished size"=189060013]

有情况是写到中途，出现该错误，经过重试机制后，会重新开始dump过程，请问是否有断点续传的机制，可以不用全部重新开始dump，因为使用的场景经常是100G-1TB的dump

lichunzhu commented 3 years ago

导出 TiDB 是有重试机制的，重试单位是一个 chunk(如果 TiDB 版本 >= 3.0 一个 chunk 是一个 region 的大小，最大 100MB 左右)。断点续传目前还没有

hu177 commented 3 years ago

导出 TiDB 是有重试机制的，重试单位是一个 chunk(如果 TiDB 版本 >= 3.0 一个 chunk 是一个 region 的大小，最大 100MB 左右)。断点续传目前还没有

请问是设置的参数有问题么 我的重试都是从开头开始重试的

lichunzhu commented 3 years ago

请问是设置的参数有问题么 我的重试都是从开头开始重试的

TiDB 版本和 dumpling 版本各自是多少呢？

hu177 commented 3 years ago

TiDB 版本和 dumpling 版本各自是多少呢？ TiDB v4.0.9 dumpling是Master分支的

lichunzhu commented 3 years ago

TiDB v4.0.9 dumpling是Master分支的

从头开始重试是说整个表重试还是整个数据块重试呢？

hu177 commented 3 years ago

TiDB v4.0.9 dumpling是Master分支的

从头开始重试是说整个表重试还是整个数据块重试呢？

整个表数据都重试了

lichunzhu commented 3 years ago

solution: Use newest dumpling, specify -r and -T argument instead of --sql argument.