Flat storage resharding: basics

[Trisfald]

I would like to get some very early feedback on the direction I'm taking for flat storage resharding.

I added few lines of code to illustrate the general idea:

• Add a new status specific to resharding in FlatStorageStatus
• FlatStorageStatus::Resharding will be triggered where there's a resharding event
• FlatStorageStatus::Resharding might be the status resumed at node start through the existing code path of flat storage creation
• A struct FlatStorageReshard will take care of doing the resharding changes for the impacted shards and finally set their status back to Ready

[wacban]

Can you confirm if this is roughly how it'll work?

• trigger
  • mark parent flat storage as ParentSplitting
  • create flat storage for both children and mark both as CreatingChild
  • send resharding request to some actor e.g. SyncJobsActor
• handle resharding request
  • iterate parent and insert to either / both children
  • mark both children as Ready
  • remove parent
• in the meantime chain is trying to add blocks to flat storage of children but since it's in CreatingChild state it stores deltas
• eventually the children flat storages are ready and the flat storage head will be moved to its usual place (somewhere close to the final block)

[Trisfald]

Can you confirm if this is roughly how it'll work?

• trigger

  • mark parent flat storage as ParentSplitting
  • create flat storage for both children and mark both as CreatingChild
  • send resharding request to some actor e.g. SyncJobsActor

• handle resharding request

  • iterate parent and insert to either / both children
  • mark both children as Ready
  • remove parent
• in the meantime chain is trying to add blocks to flat storage of children but since it's in CreatingChild state it stores deltas
• eventually the children flat storages are ready and the flat storage head will be moved to its usual place (somewhere close to the final block)

Yes, correct! I'm not sure about few implementation details, like making or extending an actor or using threads / rayon. Is the actor the usual way to go?

[Trisfald]

Thanks for the feedback, I'll close this draft and reopen once I have more code committed