enhance: accelerate `load-backup` command speed

milvus-io / birdwatcher

Apache License 2.0

43 stars 32 forks source link

enhance: accelerate `load-backup` command speed #233

Closed congqixia closed 8 months ago

congqixia commented 8 months ago

Previously the load-backup command cost lots of time if there are numerous kv pairs in backup file.

This PR accelerate this procedure with

Batching the KV entries before writing into embed etcd
Use multiple workers to execute the put transaction

For 110k kv backup file, this could improve load time from about 5.5 minutes to around 15 seconds.

are numerous

sre-ci-robot commented 8 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/milvus-io/birdwatcher/blob/v1.0.x/OWNERS)~~ [congqixia] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

yah01 commented 8 months ago

/lgtm