search

pingcap / dumpling

Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).
Apache License 2.0
281 stars 85 forks source link

dump: support dump phase network speed limit #275

Closed feloxx closed 2 years ago

feloxx commented 3 years ago

What problem does this PR solve?

To solve the #274.

What is changed and how it works?

STEP1: 

Add the dump phase network speed limit feature.
Mainly modify the `writer` related code.

STEP2:

Modify the config code of DM later.

STEP3:

Update the document.

This PR is STEP1.

Check List

Tests

Observing whether the network traffic in Grafana is stable.

Release note

No release note

ti-chi-bot commented 3 years ago

Welcome @feloxx!

It looks like this is your first PR to pingcap/dumpling 🎉.

I'm the bot to help you request reviewers, add labels and more, See available commands.

We want to make sure your contribution gets all the attention it needs!



Thank you, and welcome to pingcap/dumpling. :smiley:

CLAassistant commented 3 years ago

CLA assistant check
All committers have signed the CLA.

lichunzhu commented 3 years ago

Could you please help sign the CLA? I will review this later.

recall704 commented 3 years ago

I also need this feature，but i think "golang.org/x/time/rate" is better than ticker sleep.

feloxx commented 3 years ago

I am not sure whether the output log of my tctx method is feasible. Please help me. Thank you!

lichunzhu commented 3 years ago

I am not sure whether the output log of my tctx method is feasible. Please help me. Thank you!

So sorry for so late to review this PR! I have checked this part the use in this PR is feasible.

Besides, I have left some comments. Looking forward to your reply~

feloxx commented 3 years ago

I rewrote it using golang.org/x/time/rate, Thank you for your review.

lichunzhu commented 3 years ago

BTW, please fix the lint

kennytm commented 3 years ago

/lgtm

ti-chi-bot commented 3 years ago

[REVIEW NOTIFICATION]

This pull request has been approved by:

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment. After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing `/lgtm` in a comment. Reviewer can cancel approval by writing `/lgtm cancel` in a comment.
lance6716 commented 3 years ago

please update the usage "Dump phase network speed limit. No setting no limit.", it's better to specify the unit (byte? KB? per second) and tell this is network bandwidth limit or disk IO limit in usage

ti-chi-bot commented 3 years ago

@feloxx: PR needs rebase.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
lichunzhu commented 2 years ago

@feloxx hello, Dumpling has been moved to https://github.com/pingcap/tidb/tree/master/dumpling now. Could you please help move this PR to pingcap/tidb?