search

apache / kvrocks

Apache Kvrocks is a distributed key value NoSQL database that uses RocksDB as storage engine and is compatible with Redis protocol.
https://kvrocks.apache.org/
Apache License 2.0
3.49k stars 456 forks source link

Refactor the data migration code #2077

Open caipengbo opened 8 months ago

caipengbo commented 8 months ago

Search before asking

Motivation

Now that we have support for DBIterator and WALIterator, we can refactor migration code based on redis-command using these two classes. Also, now that we support two different migration types, we can make the code base about migration a bit cleaner than it is now.

The unit tests could also be clearer.

Solution

No response

Are you willing to submit a PR?

wsehjk commented 7 months ago

Hi, i'm interested in this issue, could you please assign this to me?

tisonkun commented 7 months ago

@wsehjk Assigned. Go ahead :D

Based on my experience, you can share your opinion on this issue and how you'd like to implement it. If it's trivial, directly sending a patch is good.

Barely asking for an assignment without any thoughts or progress attempts often ends with giving up.

git-hulk commented 7 months ago

@caipengbo Would you mind giving some input about what you plan to improve?

caipengbo commented 7 months ago

@git-hulk This is my personal opinion for reference only:

  1. decouple the iteration logic from the send logic:
    • use DBIterator and WALIterator for data iteration
    • use a Sender helper class for sending data and rate limit
  2. Make the two migration types more uniform
  3. Refactor Golang unit tests about data migration to make them more concise and faster to run
  4. Other minor change:
    • MigrationJob stores what is really needed
    • ...

As @tisonkun said, you can also share some of your plans and we can discuss them together @wsehjk .

git-hulk commented 7 months ago

@caipengbo Thanks for your input.

wsehjk commented 7 months ago

@git-hulk This is my personal opinion for reference only:

  1. decouple the iteration logic from the send logic:

    • use DBIterator and WALIterator for data iteration
    • use a Sender helper class for sending data and rate limit
  2. Make the two migration types more uniform
  3. Refactor Golang unit tests about data migration to make them more concise and faster to run

  4. Other minor change:

    • MigrationJob stores what is really needed
    • ...

As @tisonkun said, you can also share some of your plans and we can discuss them together @wsehjk .

Thank you for the info, really helpful, and i will walk through the code and share my thought before developing