ptjhuang
commented
1 year ago We are also battling with this ever since going to production, and has become more than a nuisance now that the system is getting heavier use.
TLDR; the latest experiment we're testing is to use linearizable read concern when querying with EventsByTag or AllEvents. Would appreciate if @jaydeboer and others could test if this is a solution for them too.
My experiments so far seem to suggest this is down to how writes take place, and what data is returned by queries from MongoDB.
There is a danger zone around when documents have just been sent for writing. It seems that after the BsonTimestamp (Ordering field) is assigned to documents to be written, the order in which those documents become available for querying is not necessarily the same as the timestamp.
What that means is that when querying for events with Ordering >= 100, you may get event 100, 102, but not 101 which is in-flight. The code will of course next ask for Ordering >= 103, which means 101 has been skipped. But if you replay from 100 again some time later, you will now see your skipped event 101.
Here's how we came to this hypothesis:
- Problem unreproducible when we deliberately look at
Ordering < (now - 1 second)instead of< Int64.MaxValue - Problem unreproducible when using linearizable read concern, which waits for concurrent writes to finish before returning, see this discussion
- Problem unreproducible when rehydrating actors - replay doesn't usually happens immediately after event writes
- Problem unreproducible when concurrently persisting and querying with
AllEventsfrom a single actor - writes are serialized as per actor characteristic.WriteMessagesAsyncconcurrently writes messages if coming from multiple actors. - Problem unreproducible when we have a slow subscriber querying with
AllEvents- writes are well and truly settled into the database by the time the query gets to it
For completeness, performance characteristics from our testing with 4M messages (200 actors, 20k msgs each), on a single machine dev environment, MongoDb in docker single-node replica set:
- Local read concern (currently the default) - causes skipping (~3500 msgs/sec written then read)
- Snapshot read concern - causes skipping at 3.4 mins (~3500 msgs/sec written then read)
- Majority read concern - causes skipping 19 mins although not frequently (~3000 msgs/sec written then read)
- Linearizable read concern - OK (~25 mins, 2800 msgs/sec written then read)
- Do not look at T-1s + local read concern - OK (~ 24 mins, 2800 msgs/sec written then read)
There's definitely a drop in performance, but I'd say it's better than skipping events.
You can change the read concern via the connection string.
Test harness - I've omitted the test actor and events - they're super simple, does nothing but calls Persist() to write the event.
public async Task All_events_should_project_correctly() {
var actorCount = 200;
var msgCount = 20_000;
Log.Info($"TenantId {TestTenantId}");
// Setup actors
Func<int, WidgetId> getActorId = actorId => WidgetId.With(Guid.Parse($"00000000-0000-0000-0000-{actorId:000000000000}"));
Func<WidgetId, string> toDisplay = id => $"widget_{id.GetGuid().ToString("n").TrimStart('0')}";
var actors = Enumerable.Range(1, actorCount)
.Select(i => Sys.ActorOf(Props.Create(() => new WidgetActor(getActorId(i), TestTenantId))))
.ToArray();
// Concurrently issue command to all actors to cause concurrent write
_ = Source.UnfoldInfinite(0, msgId => (msgId + 1, msgId + 1))
.Take(msgCount)
.SelectAsync(actorCount, async msgId => {
return await Task.WhenAll(actors.Select((actor, i) => {
var cmd = new WidgetCommand(getActorId(i+1),
BatchTestFailMode.None,
null);
return actor.Ask<WidgetCommandResult>(cmd);
}));
})
.SelectMany(results => results)
.RunWith(Sink.Ignore<WidgetCommandResult>(), Sys.Materializer());
// Subscribe using AllEvents
await ((Source<long, NotUsed>)MongoJournalQuery
.AllEvents(Offset.NoOffset())
.GroupBy(int.MaxValue, env => ((ICommittedEvent<WidgetEvent>)env.Event).Data.Id)
.Scan((-1L, ImmutableArray<long>.Empty), (prev, currEnv) => {
var curr = (ICommittedEvent<WidgetEvent>)currEnv.Event;
var (prevSeq, prevList) = prev;
var currSeq = curr.Data.Sequence;
if (currSeq != prevSeq + 1) {
throw new Exception($"Received {currSeq} from {toDisplay(curr.Data.Id)}, expecting {string.Join("; ", prevList.TakeLast(5))}; [{prevSeq + 1}] ({currEnv.Offset.ToInt64()})");
}
return (prevSeq + 1, prevList.Add(currSeq));
})
.Select(t => 0L)
.MergeSubstreams())
.Take((msgCount + 1) * actorCount) // Scan produces 1 extra initial message
.IdleTimeout(TimeSpan.FromSeconds(300))
.RunWith(Sink.Ignore<long>(), Sys.Materializer());
}
jaydeboer
Aaronontheweb
Arkatufus
Version Information Version of Akka.NET?
Describe the bug When there are a fair number of writes and reads happening at the same time, an event will be skipped. I have tested this against MongoDB running as a single node cluster running in Docker. I have reproduced the same behavior with EventsByTag and AllEvents queries. It may take many thousands of events for one to be skipped.
To Reproduce I have a sink setup as follows:
Expected behavior Each of the events would be sent to the subscriber.
Actual behavior Some go missing.
Environment .NET 7 Windows 11 and MacOS both exhibit the same behavior.