Open mariomastrodicasa opened 2 years ago
The benchmark cannot be built with an absolute vision. What it can done is to have some function that executes in the same way, but use a different underlying mechanism: i.e. use KNet in some function and use e.g. Confluent.Kafka in other. Pairing this two executions it is possible to compare both implementations. Anyway it is mandatory to use configurations which, more or less, create similar environments: if the configuration is in one case, or in another, is optimized the comparison has no real meaning. The previous consideration comes from the number of possible options can be used when a consumer/producer is allocated, how the underlying mechanism manages message batching both in receive mode (consume) and send mode (produce).
Maybe a new option shall be added, or a value shall be calculated, about the Average, stdev and CV without the maximum and minimum values in the series. Max and Min in a series can be affected from spurius conditions and can affect the other measurements: try to remove them and calculate the values.
Try to integrate BenchmarkDotNet (at https://github.com/dotnet/BenchmarkDotNet)
@masesdevelopers: Add a new benchmark which measures the time elapsed from the moment where a record is sent from the producer to the reception of the same record from a consumer: it is like a roundtrip. I think this benchmark can measure both producer and consumer performance, maybe it is possible to use https://github.com/masesgroup/KNet/issues/53#issuecomment-1126036921
Reopen to update statistics with latest versions of KNet and Confluent.Kafka
Many KNet operations can, or cannot, be impacted during JNI operations because data can, or cannot, be the copy of the data available in the JVM.
As stated in https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/functions.html JVM can decide to copy, or pin, data depending on its internal implementation.
The data transfer used in KNet is based on JNI and the results of some benchmarks seems impacted from the following hypothesis:
Current benchmarks are based on byte
array to reduce exogenous interference (like the implementation of the serializers) and in general KNet specific implementation (KNetConsumer, KNetProducer, KNet Streams SDK, etc) uses byte
array.
Maybe a better data exchange can be obtained reducing the number of array copies done during execution. An issue will be opened to investigate on this possible evolution, meanwhile this issue is reopened.
The information traverse the CLR-JVM boundary many times reducing the speed.
The local serializers stub invokes remote serializers: each time there is a conversion the JVM is impacted like in https://github.com/masesgroup/KNet/blob/721fe074212d4e221630853368beaab6a11bf120/src/net/KNet/Specific/Serialization/KNetSerialization.cs#L240 or https://github.com/masesgroup/KNet/blob/721fe074212d4e221630853368beaab6a11bf120/src/net/KNet/Specific/Serialization/KNetSerialization.cs#L347
Each conversion moves data between CLR and JVM:
bool
, int
, double
, etc) is sent to JVM byte array
byte array
is returned to the caller that insert it within e.g. a ProducerRecord
ProducerRecord
send back again the array to the JVMThe same happens in opposite with ConsumerRecord
:
ConsumerRecord
byte array
is requested to the JVM which send it back to the CLRbyte array
back again to the JVMbool
, int
, double
, etc)Add a specific workflow to execute benchmarks and reports statistics on the repo
Is your feature request related to a problem? Please describe. The project needs a benchmark to verify its performance.
Describe the solution you'd like Add some benchmark associated to the produce/consume API calls to verify performance and identify possible bottlenecks.
Describe alternatives you've considered Introduce some counter or other mechanism to measure time like
Stopwatch
.Additional context N/A