search

pingcap / tidb-lightning

This repository has been moved to https://github.com/pingcap/br
Apache License 2.0
142 stars 66 forks source link

Support load compressed files from Mydumper #260

Open shonge opened 4 years ago

shonge commented 4 years ago

Feature Request

Is your feature request related to a problem? Please describe:

Lightning can't load compressed files when using mydumper --compress

Backup files:

-rw-r--r--. 1 root root   85 Jan 16 11:42 app-schema-create.sql.gz
-rw-r--r--. 1 root root  213 Jan 16 11:42 app.mytable_tbl-schema.sql.gz
-rw-r--r--. 1 root root   98 Jan 16 11:42 app.mytable_tbl.sql.gz
-rw-r--r--. 1 root root  146 Jan 16 11:42 metadata
-rw-r--r--. 1 root root   87 Jan 16 11:42 mysql-schema-create.sql.gz
-rw-r--r--. 1 root root  167 Jan 16 11:42 mysql.GLOBAL_VARIABLES-schema.sql.gz
-rw-r--r--. 1 root root 5943 Jan 16 11:42 mysql.GLOBAL_VARIABLES.sql.gz
-rw-r--r--. 1 root root  238 Jan 16 11:42 mysql.bind_info-schema.sql.gz
-rw-r--r--. 1 root root  253 Jan 16 11:42 mysql.columns_priv-schema.sql.gz
-rw-r--r--. 1 root root  307 Jan 16 11:42 mysql.db-schema.sql.gz
-rw-r--r--. 1 root root  204 Jan 16 11:42 mysql.default_roles-schema.sql.gz
-rw-r--r--. 1 root root  142 Jan 16 11:42 mysql.expr_pushdown_blacklist-schema.sql.gz
-rw-r--r--. 1 root root  260 Jan 16 11:42 mysql.gc_delete_range-schema.sql.gz
-rw-r--r--. 1 root root  263 Jan 16 11:42 mysql.gc_delete_range_done-schema.sql.gz
-rw-r--r--. 1 root root  244 Jan 16 11:42 mysql.help_topic-schema.sql.gz
-rw-r--r--. 1 root root  136 Jan 16 11:42 mysql.opt_rule_blacklist-schema.sql.gz
-rw-r--r--. 1 root root  243 Jan 16 11:42 mysql.role_edges-schema.sql.gz
-rw-r--r--. 1 root root  232 Jan 16 11:42 mysql.stats_buckets-schema.sql.gz
-rw-r--r--. 1 root root  188 Jan 16 11:42 mysql.stats_feedback-schema.sql.gz
-rw-r--r--. 1 root root  304 Jan 16 11:42 mysql.stats_histograms-schema.sql.gz
-rw-r--r--. 1 root root  141 Jan 16 11:42 mysql.stats_histograms.sql.gz
-rw-r--r--. 1 root root  212 Jan 16 11:42 mysql.stats_meta-schema.sql.gz
-rw-r--r--. 1 root root   81 Jan 16 11:42 mysql.stats_meta.sql.gz
-rw-r--r--. 1 root root  206 Jan 16 11:42 mysql.stats_top_n-schema.sql.gz
-rw-r--r--. 1 root root  311 Jan 16 11:42 mysql.tables_priv-schema.sql.gz
-rw-r--r--. 1 root root  172 Jan 16 11:42 mysql.tidb-schema.sql.gz
-rw-r--r--. 1 root root  617 Jan 16 11:42 mysql.tidb.sql.gz
-rw-r--r--. 1 root root  354 Jan 16 11:42 mysql.user-schema.sql.gz
-rw-r--r--. 1 root root  167 Jan 16 11:42 mysql.user.sql.gz
-rw-r--r--. 1 root root   86 Jan 16 11:42 test-schema-create.sql.gz

Ligntning logs:

[2020/01/16 12:41:29.550 +00:00] [INFO] [lightning.go:194] ["load data source start"]
sync log failed sync /dev/stdout: invalid argument
[2020/01/16 12:41:29.551 +00:00] [ERROR] [lightning.go:197] ["load data source failed"] [takeTime=421.42µs] [error="missing {schema}-schema-create.sql"]
[2020/01/16 12:41:29.551 +00:00] [ERROR] [main.go:59] ["tidb lightning encountered error"] [error="missing {schema}-schema-create.sql"] [errorVerbose="missing {schema}-schema-create.sql\ngithub.com/pingcap/tidb-lightning/lightning/mydump.(*mdLoaderSetup).setup\n\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-lightning/lightning/mydump/loader.go:176\ngithub.com/pingcap/tidb-lightning/lightning/mydump.NewMyDumpLoader\n\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-lightning/lightning/mydump/loader.go:105\ngithub.com/pingcap/tidb-lightning/lightning.(*Lightning).run\n\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-lightning/lightning/lightning.go:196\ngithub.com/pingcap/tidb-lightning/lightning.(*Lightning).RunOnce\n\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-lightning/lightning/lightning.go:138\nmain.main\n\t/home/jenkins/agent/workspace/release_tidb_3.0/go/src/github.com/pingcap/tidb-lightning/cmd/tidb-lightning/main.go:56\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337"]

Describe the feature you'd like:

Can load the compressed data files from mydumper.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Optimization:

shonge commented 4 years ago

Now, we use gzip -d *.gz command to workaround.