Proposal: reminders

[Horusiath]

Motivation

When it comes to messages that can be scheduled to be send to an actor somewhere in the future, we could split them into two categories:

1. High-frequency timers: those are messages scheduled for sub-second intervals, that need to be delivered fast in performant way. Their downside is that they usually are not reliable for longer periods of time.
2. Long-living delays (reminders): those are messages scheduled somewhere in the future (we speak about minutes, hours or even days). Because of that nature, schedule events needs to be persisted.

Currently we have pt. 1 solved as part of ActorSystems scheduler. Pt 2 can be solved using Akka.Quartz plugin, but there's a problem: Quartz is quite heavy, and ensuring persistence usually is associated with some cost (serveral extra tables needs to be created). Another problem is lack of cluster awareness.

Proposal

The API could be simplistic, only schedule delayed messages. If user needs to schedule periodical tasks, he can always reschedule them after receiving previous one. No need to complicate things. Example:

var schedule = new Reminder.Schedule(
    key: "key", 
    recipient: recipientActorRef.Path, 
    message: new MyMessage(), 
    triggerDate: DateTime.UtcNow + TimeSpan.FromHours(1));
var reminderRef = Reminder.Get(system).ReminderRef;
var reply = await reminderRef.Ask(schedule); // wait for ACK

• Here key is unique identifier for target reminded call. It could be used i.e. to track its status or cancellation.
• Recipient is ActorPath: same thing as with at-least-once delivery. Since we can call it weeks in the future, we don't need to rely on actor ref here.

Additional thing to think about is delivery guarantee: are we going to simply push the message to a provided actor path or expect some kind of response before completing reminder request? To achieve confirmed reminders we can always combine them with at-least-once delivery actors.

Solution 1: eventsourced reminders

We could easily attach extra Akka.NET extension as part of Akka.Persistence library, that would expose a dedicated actor, which could utilize eventsourcing capabilities of actors to schedule long-living jobs (like in case of cluster sharding, they could have configurable journal ids and snapshot store ids). The biggest problem I see there is that Akka.Persistence actors don't work in multi-writer scenarios. This enforces reminders to use coordinators in case of cluster-wide reminders.

Solution 2: CRDT-based reminders

Since we already have a distributed, replicated data types in akka, we could make use of them. This way we could have cluster-wide available reminders (so even if one machine dies, the reminder can still be picked up by another one). This however brings two problems:

• Actors addresation: recipient of the message must be addressable from whatever node will pick up the reminder (I've already proposed universal actor addressation semantic on akka issue tracker). At this point in time it should be everliving actor (i.e. shard region in case of cluster sharding).
• Actors living on different nodes would need to coordinate, which one is going to pickup reminder, so the associated message won't be picked up and send multiple times. The simplest solution would be to allow only one actor (i.e. role leader) to work as sender, but I'm opened to suggestions.

Less immediate problem is that our binary format necessary for ddata persistence is still not set in stone.

[Aaronontheweb]

@Horusiath With respect to which implementation would be preferable.... Do you anticipate many use cases for reminders without Akka.Cluster? That would tip the scale towards piggy-backing on top of Akka.Persistence, IMHO.