Particular / ServiceControl

Backend for ServiceInsight and ServicePulse
https://docs.particular.net/servicecontrol/
Other
51 stars 47 forks source link

Test performance of session-based audit ingestion #3208

Open mikeminutillo opened 1 year ago

mikeminutillo commented 1 year ago

The RavenDB 3.5 persistence audit ingestion is based on RavenDB bulk-inserts. When we developed the RavenDB 5 persistence, we also based it on RavenDB bulk inserts.

We have seen some evidence that a session-based insert could work just as well.

3096 contains a spike showing the approach. It should be tested to see if it performs significantly better than the bulk insert based approach.

mikeminutillo commented 1 year ago

In a call with Hibernating Rhinos they also suggested moving away from BulkInsert for our use case. BulkInserts are faster in RavenDB 3.5 because RavenDB acquired a global lock on the database during session operations. With ravenDB 5 we would be better off with a pattern of interleaved operations feeding from the transport and shuffling to RavenDB. Our batch sizes are too small to benefit from bulk insert. The example given was that we should split our 'concurrency' in half and then have a few ravendb sessions that store multiple operations and then call SaveChanges, like this:

var batchSize = maxConcurrency / 2;
while(channel.TryRead() && batchSize < maxBatchSize) 
{
    async Task Save(Messages messages)
    {
       using var session = documentStore.OpenAsyncSession();
       foreach(var message in messages) {
          await session.Store(message);
       }
       await session.SaveChangesAsync();
    }

    // Interleave Save calls (two max at the same time in the above example)
}