I’m currently working on building an in-memory database that can handle datasets larger than the available RAM.
For persistence, I’m using Faster Log, which seems very promising. The concurrency control relies on the ReaderWriterLockSlim, which keeps the implementation simple, particularly for synchronous APIs (without using async/await).
The challenge appears when considering how to fetch data from the log once the data record has been flushed to disk (to maintain a small memory footprint). I noticed that the log only provides an asynchronous method for this:
However, this creates an issue for my implementation, which relies on ReaderWriterLockSlim. It breaks serializable execution because I can’t use async/await within a transaction.
Having a synchronous version of log.ReadAsync would be incredibly helpful. That way, during a transaction, I could verify that a record is missing from RAM but still fetch it from disk using its address.
Maybe I can use log.ReadAsync(iter.CurrentAddress).GetAwaiter().GetResult(), but I am not sure about deadlocks that may happen with such an approach.
AntyaDev
commented
3 days ago
Thank you so much for this amazing project!
I’m currently working on building an in-memory database that can handle datasets larger than the available RAM. For persistence, I’m using Faster Log, which seems very promising. The concurrency control relies on the ReaderWriterLockSlim, which keeps the implementation simple, particularly for synchronous APIs (without using async/await).
The challenge appears when considering how to fetch data from the log once the data record has been flushed to disk (to maintain a small memory footprint). I noticed that the log only provides an asynchronous method for this:
However, this creates an issue for my implementation, which relies on ReaderWriterLockSlim. It breaks serializable execution because I can’t use async/await within a transaction.
Having a synchronous version of
log.ReadAsyncwould be incredibly helpful. That way, during a transaction, I could verify that a record is missing from RAM but still fetch it from disk using its address.Maybe I can use
log.ReadAsync(iter.CurrentAddress).GetAwaiter().GetResult(), but I am not sure about deadlocks that may happen with such an approach.I would appreciate any suggestions.