Add FASTER log broker implementation

[mookid8000]

The current file-based log is based on Kafkaesque

Topos could probably see a significant increase in performance by using FASTER log once it has been made available.

[badrishc]

Sounds like an interesting use case. We are open to design suggestions or ideas as we finalize the interface and features for the component.

[mookid8000]

Sounds like an interesting use case.

Yes, I think it is an interesting use case 🙂 Topos is basically an abstraction over the "log-based message broker" formula, which makes it easy to e.g. write code that it portable and can be running on either a full Kafka cluster or by using the file system as the transport (that's what I am using it for myself: I have a cloud-hosted multi-tenant SaaS product, which my customers can also choose to run on-prem on their own hardware).

Right now, the on-prem configuration runs on aforementioned Kafkaesque, which basically provides the following API:

var logDirectory = new LogDirectory(@"C:\data\kafkaesque");

// hold on to this bad boy until your application shuts down
var logWriter = logDirectory.GetWriter();

// use logWriter concurrently from as many threads as you like to append byte chunks to the log
await logWriter.WriteAsync(new byte[] {1, 2, 3});

for a writer (of which there can be only one, as it takes write locks on files), and

var logDirectory = new LogDirectory(@"C:\data\kafkaesque");

var logReader = logDirectory.GetReader();

foreach (var logEvent in logReader.Read())
{
    var bytes = logEvent.Data;

    // process the bytes here
}

for readers, of which there can be as many as one wants.

Each read log event then comes with position information, making it possible for readers to resume reading from the last event they read.

We are open to design suggestions or ideas as we finalize the interface and features for the component.

If FASTER log will provide a similar interface (i.e. the ability to append byte chunks and read them back concurrently, possibly resuming from a known position) – and I bet it will – then I'm super happy 👍 🤠

[mookid8000]

Closing this, because Topos has had the FASTER-based broker implementation for quite some time now.

https://github.com/mookid8000/Topos/tree/master/Topos.Faster