microsoft / FASTER

Fast persistent recoverable log and key-value store + cache, in C# and C++.
https://aka.ms/FASTER
MIT License
6.29k stars 563 forks source link

FasterKV: Is there a way to wait for flush globally including all pending operations? #879

Closed AqlaSolutions closed 9 months ago

AqlaSolutions commented 11 months ago

At some point we need to ensure that everything we've added to FasterKV so far is persisted including all Upsert calls issued previously. We have many FasterKV parallel sessions which do Upserts. Is there a way to globally wait for all pending IO to complete before initiating a checkpoint?

We don't want to separately call CompletePending on each session, there are too many of them and it would be a huge synchronization overhead (because we use actor model and each actor has its own session).

badrishc commented 9 months ago

We don't have this option. It might be better to keep a shared pool of sessions, borrow from the pool, and return to the pool within each actor grain. See this for example:

https://github.com/microsoft/FASTER/blob/main/cs/playground/AsyncStress/SerializedFasterWrapper.cs#L16