search

pingcap / dumpling

Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).
Apache License 2.0
280 stars 85 forks source link

Dump with `--sql` fails with i/o timeout after 15m #317

Closed dveeden closed 2 years ago

dveeden commented 3 years ago

Bug Report

  1. What did you do? If possible, provide a recipe for reproducing the error.

Running a dumpling --sql $statement

  1. What did you expect to see?
...
[2021/07/23 10:30:15.947 +02:00] [INFO] [main.go:81] ["dump data successfully, dumpling will exit now"]
  1. What did you see instead?
...
[mysql] 2021/07/23 10:26:21 packets.go:36: read tcp 127.0.0.1:54624->127.0.0.1:8022: i/o timeout

  1. Versions of the cluster

    • Dumpling version (run dumpling -V):

      Release version: v5.0.1
Git commit hash: 4cb115746bb658b6d1a12c0e49932bfd3a08afac

    • Source database version (execute SELECT version(); in a MySQL client):

      8.0.22

To reproduce:

dumpling -P 3306 -p $pasword -h 127.0.0.1 --sql 'select sleep(180)' --read-timeout 50s

Or without --read-timeout 50s and with SELECT SLEEP(1900) (more than 15m)

This is due to hidden read timeout setting that is set to 15m by default.

        flags.Duration(flagReadTimeout, 15*time.Minute, "I/O read timeout for db connection.")
        _ = flags.MarkHidden(flagReadTimeout)

Note that for long running SQL it might be needed to set --params 'wait_timeout=3600 depending on what's set on the server side (default 28800 (8h) for MySQL).

Some possible fixes/improvements:

  1. Make the --read-timeout variable and default visible to the user
  2. Have the status watch the read timeout and give a clear error if it (almost) expires.
  3. Don't set a read timeout by default (but that could re-introduce #190 )
  4. Update the docs
fubinzh commented 3 years ago

/type bug

fubinzh commented 3 years ago

/severity moderate

lichunzhu commented 3 years ago 
[mysql] 2021/07/23 10:26:21 packets.go:36: read tcp 127.0.0.1:54624->127.0.0.1:8022: i/o timeout

This log is from github.com/go-sql-driver/mysql and we can't modify this. I prefer to add document to deal with this problem.

dveeden commented 3 years ago 
[mysql] 2021/07/23 10:26:21 packets.go:36: read tcp 127.0.0.1:54624->127.0.0.1:8022: i/o timeout

This log is from github.com/go-sql-driver/mysql and we can't modify this. I prefer to add document to deal with this problem.

We can still unhide --read-timeout and/or change the default.

lichunzhu commented 2 years ago

When the connection is broken we may not get results from the database server for a long time. I think default 15 minutes is reasonable because in a few cases we need to receive data to wait more than 15mins. We hide this parameter to make it less confusing for users. Feel free to reopen this issue if you think it's still a problem.