Long processing time with simultaneous transactions

[duizer]

The problem

I am running a test of issuing 24 production certificates. Initially, I wanted to simultaneously create 24 transactions. But processing those takes around 9 minutes (see registry-simul.txt for logs from registry server).

Changing this to issue 1 production certificate and then wait for the transaction to be committed before continuing to next production certificate, the processing time is around 2 minutes (see registry-sequential.txt for logs from registry server).

Expectaion

I would have expected the simultaneous approach to be faster, not slower.

How to reproduce

This is tested using test containers in this setup:

        IssuerKey = Algorithms.Ed25519.GenerateNewPrivateKey();

        verifierContainer = new ContainerBuilder()
                .WithImage(electricityVerifierImage)
                .WithPortBinding(GrpcPort, true)
                .WithEnvironment($"Issuers__{IssuerArea}", Convert.ToBase64String(Encoding.UTF8.GetBytes(IssuerKey.PublicKey.ExportPkixText())))
                .WithWaitStrategy(
                    Wait.ForUnixContainer()
                        .UntilPortIsAvailable(GrpcPort)
                    )
                .Build();

        registryContainer = new Lazy<IContainer>(() =>
        {
            var verifierUrl = $"http://{verifierContainer.IpAddress}:{GrpcPort}";
            return new ContainerBuilder()
                .WithImage(registryImage)
                .WithPortBinding(GrpcPort, true)
                .WithEnvironment("Verifiers__project_origin.electricity.v1", verifierUrl)
                .WithEnvironment("RegistryName", registryName)
                .WithEnvironment("IMMUTABLELOG__TYPE", "log")
                .WithEnvironment("VERIFIABLEEVENTSTORE__BATCHSIZEEXPONENT", "0")
                .WithWaitStrategy(
                    Wait.ForUnixContainer()
                        .UntilPortIsAvailable(GrpcPort)
                    )
                .Build();
        });

[wisbech]

Thank you for the information @duizer, on these very long batch wait times.

This looks to me like an under the hood problem, I only have a suspicion on the root cause, but it might the conversion between rust and c#, maybe it would be fitting to have a benchmark and have our rust shark @Quacktiamauct look a potential refactoring of the merkletree implementation.

[quackzar]

After some playing with some crude benchmarks I conclude that it isn't the inter-op layer bottlenecking the system, and not with parallelized workloads. From dotnet we can construct 100 proofs in 432 ms sequentially and in 80 ms parallel, at least on my machine. I will try to provide some benchmarking demo in the pedersen-commitment later this week. I have however only performed this natively, so if docker or anything is doing something weird then that might be it, although unlikely.

[MartinSchmidt]

I will take a look at it

[MartinSchmidt]

@duizer, I would like more info from you, which images do you use? Possible whole test file if you could.

[duizer]

@MartinSchmidt, sure. See here: https://github.com/Energinet-DataHub/energy-origin/blob/1638-new-registry-client/domains/certificates/Query.API/API.IntegrationTests/Testcontainers/RegistryFixture.cs