nkosi23
commented
1 year ago Indeed, a more detailed documentation could be useful on this point. I have found this post describing the activation logic. I do not think things have changed a lot since then so hopefully this is useful:
Let me give you an example, we have three nodes: B, C, and D. In node C activated Grain[A-1], the activation information is saved on Node B. This time the new node A joins the cluster, node A is the direct forward of B, and the information of Grain[A-1] now belongs to A. At this point, when the status of node A becomes active, node B begins to process the AddServer event and prepares to give the Grain[A-1] location data that it retains to node A. But at this point, node D generates a call to Grain[A-1], at which point node D will see if the location information is on node A, but because node B has not given the data to node A, so node D found no grain[A-1] location information, then according to the default policy, will randomly select a node to activate, Assuming node D is selected, then D will have a grain[A-1] activation. The cluster will now have two grain[A-1] activations.
I'd say if you absolutely cannot afford stale data in a specific scenario, the best approach as of today may be to always read the state from database and never rely on the cache. The difficulty here is that Orleans cannot reliably help us for example by notifying us when a duplicate activation has been resolved, since when the cluster is not stable all nodes do not necessarily have the same view of the cluster, so the winning activation may not even know that another activation existed.
Maybe using a storage-based grain directory would help with that (instead of the default in-memory directory) however the notification API does not exist for now so this may not be a solution right now.
Edit: Also I just remembered that if you use the persistence framework provided by Orleans, you get ETag checks for free, which gives you the peace of mind of knowing that if the ETag you have in-memory differs from the one in the database the transaction will be rejected and throw an exception. If you do not or cannot use Orleans' built-in persistence framework, rolling out a similar approach by hand may be a solution (or using a database such as CouchDB which automatically uses a revision sequence number similar to Etags).
As far as reads are concerned, only you can determine when reading stale data would be unacceptable for your application. My experience is that strictly instant consistency is rarely needed in practice. Also keep in my that you have the option of exposing two APIs: ReadPotentiallyStale() and CheckEtagAndReloadFromStoreIfNeededBeforeRead() depending on the consumer to avoid premature optimization.
Also, another useful piece of information is that an old documentation website says that the system will converge toward the single activation guarantee within 30 seconds depending on configuration:
In what cases can a split brain (same grain activated in multiple silos at the same time) happen?
During normal operations the Orleans runtime guarantees that each grain will have at most one instance in the cluster. The only time this guarantee can be violated is when a silo crashes or gets killed without a proper shutdown. In that case, there is a ~30 second (based on configuration) window where a grain can potentially get temporarily instantiated in more than one silo. Convergence to a single instance per grain is guaranteed, and duplicate activations will be deactivated when this window closes.
Also you can take a look at Orleans' paper for a more detailed information, however you don't need to understand it fully to be able to write your application code. You just need to consider the rare possibility of having two instances of an actor while writing your application. The persistence model guarantees that no writes to storage are blindly overwritten in such a case.
The relevant sections of the paper in question say:
3.2 Distributed Directory Many distributed systems use deterministic placement to avoid maintaining an explicit directory of the location of each component, by consistent hashing or range-based partitioning. Orleans allows completely flexible placement, keeping the location of each actor in a distributed directory. This allows the runtime more freedom in managing system resources by placing and moving actors as the load on the system changes. The Orleans directory is implemented as a one-hop distributed hash table (DHT) [17]. Each server in the cluster holds a partition of the directory, and actors are assigned to the partitions using consistent hashing. Each record in the directory maps an actor id to the location(s) of its activations. When a new activation is created, a registration request is sent to the appropriate directory partition. Similarly, when an activation is deactivated, a request is sent to the partition to unregister the activation. The single-activation constraint is enforced by the directory: if a registration request is received for a singleactivation actor that already has an activation registered, the new registration is rejected, and the address of the existing activation is returned with the rejection. Using a distributed directory for placement and routing implies an additional hop for every message, to find out the physical location of a target actor. Therefore, Orleans maintains a large local cache on every server with recently resolved actor-to-activation mappings. Each cache entry for a single-activation actor is about 80 bytes. This allows us to comfortably cache millions of entries on typical production servers. We have found in production that the cache has a very high hit ratio and is effective enough to eliminate almost completely the need for an extra hop on every message.
3.9 Eventual Consistency In failure-free times, Orleans guarantees that an actor only has a single activation. However, when failures occur, this is only guaranteed eventually. Membership is in flux after a server has failed but before its failure has been communicated to all survivors. During this period, a register-activation request may be misrouted if the sender has a stale membership view. The target of the register request will reroute the request if it is not the proper owner of the directory partition in its view. However, it may be that two activations of the same actor are registered in two different directory partitions, resulting in two activations of a singleactivation actor. In this case, once the membership has settled, one of the activations is dropped from the directory and a message is sent to its server to deactivate it. We made this tradeoff in favor of availability over consistency to ensure that applications can make progress even when membership is in flux. For most applications this “eventual single activation” semantics has been sufficient, as the situation is rare. If it is insufficient, the application can rely on external persistent storage to provide stronger data consistency. We have found that relying on recovery and reconciliation in this way is simpler, more robust, and performs better than trying to maintain absolute accuracy in the directory and strict coherence in the local directory caches.
peter-perot
ReubenBond
slawomirpiotrowski
Almost all code examples that involve stateful grains write data to the storage but avoid reading them back programmatically. The reason for this practice is that Orleans automatically reads the state on activation, and when there are no other "leaks" that change the state on the underlying database (e.g. as done by stored procedures), there is no reason to read the data back once it is cached in memory by the grain.
But now consider a split-brain scenario, where grain A reads the state at time T1, and a duplicate activation A' of the same grain (due to split-brain) gets an update call the state of which is written to the storage at time T2 > T1.
Now Orleans detects the split-brain scenario and tries to resolve it. But how? I think we need a more detailed documentation here how Orleans achieves this.
Strategy 1: If Orleans only deactivates grain A', then grain A with a stale state stays in memory, and future read-requests will all return stale data.
Strategy 2: If Orleans deactivates both grains, then the next read-request will reinstantiate the grain and re-read the latest state, so we get the correct state in this case.
Q1: What is the strategy that is implemented in Orleans?
Q2: In case the first strategy is implemented, what are the recommended countermeasures to avoid staleness? Do we need to read the state from storage on each read-request? [Clarification: Interim staleness can be tolerated, but staleness forever (in case the state is not changed again) is no option.]
There are two options for Q2 that come into my mind:
Option 1: Re-read the state when a read-request comes in and the last refresh from the storage is older than a defined period.
Option 2: Re-read the state when a read-request comes in and the last refresh from the storage is older than a defined period, but don't suspend the current call until the re-read has finished. Instead return the current potentially stale state and trigger a refresh in the background (with a non-awaited task).