Continuing a discussion from Discord, but gathering this all together to track outstanding tasks. The goal here is to reduce the overhead of appending events and Inline projections by reducing network round trips as much as possible. This is also to improve the new Quick Append event workflow and address #3310. This work is driven by scalability and performance concerns from two JasperFx clients concerned about scalability for very large workloads
Overarching changes:
Reproduce the optimization from #3290 that was backed out to make FetchForWriting() + Inline aggregates use an identity
map when loading the aggregate to avoid multiple database round trips. This time, make this behavior "opt in"
Reduce the network round trips when appending events to fetch stream state and/or aggregates
Event Appending Data Fetching
When appending events, there's potentially a series of preparatory data fetching steps that has to take place before the events
can be appended and any or all inline projections can be applied. Today this feature is very optimized for appending events to a single stream at a time -- which isn't necessarily how folks are actually using this feature today. To optimize the process -- and I'm making a very large assumption that reducing the network round trips to the database will more than offset extra in process steps -- I think we make the event appending be a multi-step process to create a new intermediate helper to enable data fetching like where the active IEventAppender and each registered Inline projection have a method something like:
public interface IEventAppendingStep
{
Task ApplyAsync(DocumentSessionBase session, CancellationToken cancellation);
}
// This would be used to register data query needs in a batch, and return a "continuation"
// that applies changes to the session later
public IEventAppendingStep FetchData(IBatchedQuery query);
There's some more detail to this so there's some interplay between Inline projections that get the stream state for you
so you don't have to make a second fetch
Tasks
[x] Recreate the QuerySession.UseIdentityMapFor<T>() changes from #3290
[x] Opt in for StoreOptions.Events.UseIdentityMapForInlineAggregates, with the default being false
[x] FetchForWriting() using an Inline aggregate should opt into using the identity map if the above setting is true
[x] Inline single stream projections should load from the identity map / session if the UseIdentityMapForInlineAggregates is turned on
[x] Test FetchForWriting() + Inline + append event has the right version afterward on document w/ IRevisioned
[x] Test FetchForWriting() + Inline + append event has the right version afterward on document w/ explicit version mapping
[x] Make 0 be the value for quick append that means "auto-version" based on the latest. Today it's 1
[Fact]
public async Task start_and_append_events_to_same_stream()
{
await using var session = theStore.LightweightSession(tenant);
session.Logger = new TestOutputMartenLogger(_testOutputHelper);
var streamId = Guid.NewGuid().ToString();
session.Events.StartStream<LoadTestInlineProjection>(streamId,new LoadTestEvent(Guid.NewGuid(), 1),
new LoadTestEvent(Guid.NewGuid(), 2), new LoadTestEvent(Guid.NewGuid(), 3));
await session.SaveChangesAsync();
_testOutputHelper.WriteLine("APPEND STARTS HERE");
session.Events.Append(streamId, new LoadTestEvent(Guid.NewGuid(), 4), new LoadTestEvent(Guid.NewGuid(), 5));
await session.SaveChangesAsync();
var doc = await session.LoadAsync<LoadTestInlineProjection>(streamId);
doc.Version.ShouldBe(5);
}
Continuing a discussion from Discord, but gathering this all together to track outstanding tasks. The goal here is to reduce the overhead of appending events and
Inline
projections by reducing network round trips as much as possible. This is also to improve the new Quick Append event workflow and address #3310. This work is driven by scalability and performance concerns from two JasperFx clients concerned about scalability for very large workloadsOverarching changes:
Event Appending Data Fetching
When appending events, there's potentially a series of preparatory data fetching steps that has to take place before the events can be appended and any or all inline projections can be applied. Today this feature is very optimized for appending events to a single stream at a time -- which isn't necessarily how folks are actually using this feature today. To optimize the process -- and I'm making a very large assumption that reducing the network round trips to the database will more than offset extra in process steps -- I think we make the event appending be a multi-step process to create a new intermediate helper to enable data fetching like where the active
IEventAppender
and each registeredInline
projection have a method something like:There's some more detail to this so there's some interplay between
Inline
projections that get the stream state for you so you don't have to make a second fetchTasks
QuerySession.UseIdentityMapFor<T>()
changes from #3290StoreOptions.Events.UseIdentityMapForInlineAggregates
, with the default being falseFetchForWriting()
using anInline
aggregate should opt into using the identity map if the above setting is trueUseIdentityMapForInlineAggregates
is turned onUpsertFunction
to reflect the aboveStreamAction.PrepareEvents()
from https://github.com/JasperFx/marten/commit/0630f5526d9ac22c76e69261e4d64b4ac4a4b0bcDocumentMapping.UseVersionFromMatchingStream
from https://github.com/JasperFx/marten/commit/0630f5526d9ac22c76e69261e4d64b4ac4a4b0bcUpsertFunction
from https://github.com/JasperFx/marten/commit/0630f5526d9ac22c76e69261e4d64b4ac4a4b0bc after the reset to 0