Add In-Memory In-Process transport

[dpavsrtrl]

I'm trying to figure out how gossip can be used as an alternative to streams or actors. Shield.Standard provides strong consistency, but for concurrent and parallel applications, eventual consistency is actually preferable (because it is naturally async). Since a transport is integral to gossip, would be nice to have an in-process one.

[jbakic]

I don't think gossip is really useful if you're not doing it over the network. Would your use case maybe be better served by using a .NET ConcurrentDictionary? You could use some type of mergeable, CRDT-like classes as values in the dictionary, and your async methods could, when they want to write something, call ConcurrentDictionary.AddOrUpdate, performing a merge in case something is already there.

Gossip protocol really shines in distributed settings, because of the risk of lost or delayed messages inherent in network communication. It will automatically route around failures, and effectively spread information around. But if you're in one process, there's no such risk of losing messages. The CRDT part of this project is what I think might be useful to you.

What do you think?

[jbakic]

P.S. I'm thinking, you could also use the GossipBackend, but just one per process. If you write a fake transport that never sends or receives anything, and has empty Servers collection, then you get the dictionary and CRDT features of this lib, with no need for any gossip. The advantage here being that, in the future, should you ever want to scale to multiple machines, you can just replace the fake transport with a real one, and the other code does not need to notice anything...

[dpavsrtrl]

I understand, and this thread is probably more to start a discussion than to request a feature.

I like the transactional nature of shielded, the shared state, and incremental updates. But I also need to be able to react to changes, and to do this asynchronously. So my idea was to use a gossipbackend per component, and an in-process transport implemented using queues.

You know, they say, share state by communicating, don't communicate by state sharing. But all too often, the messages are exactly about state changes of some component. It feels natural to just change a property and expect that change to be propagated to anyone interested, safely, asynchronously and correctly. This is from inside->out. On the other hand, what is RPC? Very often is just the desire to learn something about component. So it feels natural to just have a global state and read that property directly, instead of sending request and expecting a reply.

So what's wrong with having global state and passing it around? :innocent:

[jbakic]

I like your style 😄 Well, the whole point of Shielded is to enable easy thread-safe state sharing, so I'm definitely not against what you're saying. But it sounds like you really need just the ordinary Shielded, or Shielded.Standard if you're targeting .NET Core. Or you could even get away with just a wrapper for an ordinary ConcurrentDictionary. When reading, just read from the dict. When writing, write in the dict and fire an event on a new Task. Or fire it directly, but write the async handlers as "async void" (another thing they say you should not do 😉) and add "await Task.Yield();" on the first line, and you have a handler that's async-triggered by changes, that does not block the writing thread. And for queueing, use the BlockingCollection. It's standard, and it's fast.

I'm happy about your interest in this project, but I've got to be honest, taking the whole Shielded.Gossip machinery would probably be way too expensive for what you need.

[jbakic]

Closing the issue. In-process transport is not needed, threads of one process can access the same backend object directly.