[QUESTION] Precision on Background Distributed Operations

[Odirb]

Hi, When Fusion Cache is configured with both AllowBackgroundDistributedCacheOperations and AllowBackgroundBackplaneOperations set to false, what does "Notify on backplane" exactly means in example 1: 1) Backplane notifications have been sent to all nodes, but potentially some of them didn't receive it yet. There is a very very very small possibility that TryGetAsync return an old value for one of them 2) All nodes have received the notification, meaning TryGetAsync always return latest value

Thanks

[jodydonetti]

Hi @Odirb

Hi, When Fusion Cache is configured with both AllowBackgroundDistributedCacheOperations and AllowBackgroundBackplaneOperations set to false, what does "Notify on backplane" exactly means in example 1:

A backplane is any implementation of the IFusionCacheBackplane interface, so it means "when a method call to this is finished".

I assume you are using Redis , and with that implementation it means that the other nodes have received the notification and reacted to that.

So, in theory you can logically think of it as "all nodes have received the notification".

In practice though, and here it depends on how deep in the rabbit hole you want to go, you may want to also consider things like:

• transient network errors
• a single Redis node in a cluster is being restarted
• the entire Redis cluster is being restarted or has a problem
• all is fine on Redis land, but your app is restarting

And on and on like this, ad infinitum.

1. Backplane notifications have been sent to all nodes, but potentially some of them didn't receive it yet. There is a very very very small possibility that TryGetAsync return an old value for one of them
2. All nodes have received the notification, meaning TryGetAsync always return latest value

Eh, I get the need to know this but with distributed system this is just the way it is.

To recap, it depends on your definition of "very very very small possibility" and "always": if you mean exactly 0% chance of not having a node in-sync then no, that's not it.

But to be clear that is not a FusionCache thing or a .NET thing, it's the nature of distributed systems in general, there's no such thing as "always" and "never" for basically anything.

Hope this helps, let me know!

[Odirb]

Hi @jodydonetti, I actually use Redis for the distributed cache. My concerns were about the behavior under normal conditions, I understand that there are no guarantees under degraded conditions. Thanks a lot for your detailed answer.

[jodydonetti]

You're welcome! Another point of interest to think about is that normally with Redis pub/sub the PUBLISH command returns the number of nodes that received the message, but... in reality not so much in Redis is in cluster mode:

Integer reply: the number of clients that the message was sent to. Note that in a Redis Cluster, only clients that are connected to the same node as the publishing client are included in the count.

So yeah, it's complicated 😅

Anyway: the short version is that yes, in normal conditions you can reasonably think of it as being "immediate".