“Topic” filtering for efficient event bus message routing

[markhammond]

Hi @akade

I’m looking to build a type of event bus, with less than 100K distinct event topics. Each receiver is long-lived. Topics are structs rather than plain strings*

The unique set of topics and set of receivers are generally stable (both may increase over time, but this is a rare occurrence).

I was thinking that during registration each receiver could create their index. The number of distinct indices will be small as most receivers are interested in the same set of topics. And associate each index object with a list of receivers.

When processing a new message, rather than broadcasting and having each receiver perform filtering I can instead iterate over indexes which “match” the message topic.

I note the public API does not currently permit enumerating the frozen collection of indexes and was wondering what you think of this potential event bus topic filtering use case.

Many thanks, Mark.

• I could potentially encode structs as strings but the representation becomes unwieldy as structs embed enum flag bit masks, etc.

[akade]

Hi Mark

I'm not really sure if I understood your goal correctly, couldn't you do something along the following lines:

IndexedSet<Subscription> set = ...
                                 .WithIndex(x => x.Topic).Build();

// raising events
foreach(Subscription sub in set.Where(x => x.Topic))
    sub.Receiver.Handle(...);

record class Subscription(Topic Topic, IReceiver receiver);
record struct Topic(...);

I wrote it on my phone, so might be a bit shaky. But my idea would be using an index where a key (the topic) can have multiple values (the receivers). On a side note, just in case, make sure that your struct either fully implementing GetHashCode & Equals or using a record that will do it for you.

Wouldn't that work for you? I'm not quite sure where you'd need to know the names of the indices?

Scaling wise, there is a system that has xlose to 1M items in production that works just fine. Make sure to use the concurrent set if you need thread safety (though you'll be paying materialization costs).