Whom/When to call deltaCRDT.resetDelta()?

[yim-lee]

We had an outstanding TODO and the topic was brought up again in https://github.com/apple/swift-distributed-actors/pull/210.

resetDelta should be called some time after the delta CRDT has propagated its changes:

• By the replicator immediately after it receives LocalCommand.write for the CRDT and direct replication completes successfully (i.e., we trust gossip to do the right thing)
• By someone after gossip is done and everyone knows about the changes

Some things to keep in mind:

• Replicator has its own copy of the CRDT. If a CRDT has multiple owners then all of their changes are aggregated by their local replicator.
• Direct replication sends changes of CRDT belonging to a specific owner to other replicators in the cluster. It does NOT send the delta of local replicator's copy of the CRDT.
• Gossip (not yet implemented) should send delta of replicator's copy of the CRDT.
• We want to send changes in batches to reduce the number of messages.

One approach might be:

1. Local replicator tells local ActorOwned CRDT to resetDelta after successful direct replication
2. Local replicator calls resetDelta on its copy of CRDT after successful gossip

More thoughts in https://github.com/apple/swift-distributed-actors/pull/210#issuecomment-546013992

[ktoso]

I think I know how we'll tackle this thanks to recent CRDT Gossip Replicator work.

Notes:

• Deltas don't matter as much locally; the crdts today are value types anyway, so we can send them as-is.
• Biggest area where it matters is the CRDT gossiper; The gossiper will keep a list of [target: vv of latest known deltas], we always send it the deltas when we perform updates
• whenever we gossip to a target, we take as many deltas as we have stored and we know the target has not seen them yet (we'll have to learn what it has seen from incoming gossips)
• we can resetDelta when we hand over the delta to the gossiper I believe;
• direct local writes therefore could perhaps then be implemented as "write the specific delta that this write has caused"?

General idea... More digging soon

[yim-lee]

direct local writes therefore could perhaps then be implemented as "write the specific delta that this write has caused"

Right, I think this echoes with the original thinking as well:

Direct replication sends changes of CRDT belonging to a specific owner to other replicators in the cluster. It does NOT send the delta of local replicator's copy of the CRDT.

---

whenever we gossip to a target, we take as many deltas as we have stored and we know the target has not seen them yet

we can resetDelta when we hand over the delta to the gossiper I believe;

👍 I guess this means gossip sends a batch of deltas (which is the behavior we want) and not after every single delta?

[ktoso]

👍 I guess this means gossip sends a batch of deltas (which is the behavior we want) and not after every single delta?

Yup, we get batching there :-)