Async Projection skips events

[a-shtifanov-laya]

Hello Dear Community!

So, after some discussion on discord and some playing around i think i was able to reproduce the issue.

Short description of a problem: all events are saved correctly, but sometimes some events are not applied to async projections.

I've forked oskardudycz/EventSourcing.NetCore repo and used Helpdesk example. I've added a simple async projection and commented out all other projections from that sample project.

public record IncidentProjection(
    Guid Id,
    long Version,
    long Sequence,
    IncidentStatus Status,
    Guid? CustomerId
)
{
    public static IncidentProjection Create(IEvent<IncidentLogged> e) =>
        new(default!, e.Version, e.Sequence, IncidentStatus.Pending, e.Data.CustomerId);
    public static IncidentProjection Create(IEvent<IncidentClosed> e) =>
        new(default!, e.Version, e.Sequence, IncidentStatus.Closed, null);
    public bool ShouldDelete(IncidentClosed _) => true;
}

and added a nbomber load test to post around 50-100k of new Incidents that should produce same amount of IncidentProjections. So, just 1 aggregate and just 1 async projection is involved. The problem is that sometimes it skips a couple of events (1-2).

Marten version used: 7.20

Steps to reproduce:

1. checkout my fork/branch https://github.com/a-shtifanov-laya/marten-concurrency-bug/tree/bug/skipped-projections
2. navigate to Sample\Helpdesk and start DB docker compose up -d postgres
3. navigate to Sample\Helpdesk\Helpdesk.Api and start an API donet run -c Release
4. navigate to Sample\Helpdesk\LoadTest and start nbomber dotnet run -c Release
   • it will run for 3 minutes.. can be changed in Sample\Helpdesk\LoadTest\Program.cs .WithLoadSimulations(Simulation.KeepConstant(10, TimeSpan.FromMinutes(3)))
5. you will see in console output smth like this

if it works fine on your machine (it will log OK: ...), try a couple of times and/or maybe increasing number of simulations/minutes can help i.e. .WithLoadSimulations(Simulation.KeepConstant(10, TimeSpan.FromMinutes(5)))

Hope it can help you point out what is wrong with the code or it is really some internal issue.. Thanks!

[a-shtifanov-laya]

i just tried also the latest version 7.24 because of https://github.com/JasperFx/marten/pull/3311 but the problem was still present

then i tried also to activate EventAppendMode.Quick (with 7.24).. and wow - it really boosts the performance 😍 and i cannot reproduce the problem! tried 5 times, even with 200k of events! interesting is if this is really the solution for it or it just hid/postponed the issue

[jeremydmiller]

@a-shtifanov-laya 7.25 will have the necessary improvements to the quick append that that's going to be the new recommendation for heavy loads. That might just be the default behavior in Marten 8. I think we should pull in your load testing at some point, but I'm actually going to close this after some documentation for the next release about performance.

There's some other performance stuff coming in 7.25 as well. Thank you so much for testing out the quick append work.