search

kruize / autotune-results

Recommendations and Results from Autotune
Apache License 2.0
3 stars 6 forks source link

Discuss TechEmpower Round21 results #32

Open kusumachalasani opened 2 years ago

kusumachalasani commented 2 years ago

Measurement process: https://github.com/kruize/autotune-results/pull/31

Below are the factors considered while running the benchmark in an autotune experiment.
- Repetability
- Convergence
- Reproducibility

Above factors are measured using the below process in an experiment:
1. Each Autotune experiment is usually composed of 100 trials.
2. Each Trial tests a specific config from HPO.
3. Each trial runs the benchmark with multiple iterations. The benchmark container gets re-deployed at the start of each iteration.
4. Each iteration in a trial includes warmup and measurement cycles. Duration of warmup cycles is based on pre-run data from the benchmark.
5. For each trial, measure convergence in the benchmark data by calculating the confidence interval by using T-distribution for each metric.
6. Calculates the min, max, mean and percentile info for the metrics.

Initial discussion to generate the config: https://github.com/kruize/autotune-results/issues/28 Tunables of the experiment: https://github.com/kusumachalasani/autotune-results-1/blob/TFB_R21/techempower/experiment-16/benchmark.yaml

Initial results for the TFB Round21 for Throughput improvements: https://github.com/kruize/autotune-results/pull/30/ More Experiments with Quarkus v2.9.1.F: In progress

Configurations of baseline and Autotune:

Baseline1 Configuration:

-server -Dquarkus.vertx.prefer-native-transport=true -XX:-StackTraceInThrowable -Dquarkus.http.accept-backlog=-1 -Dio.netty.buffer.checkBounds=false -Dio.netty.buffer.checkAccessible=false -Djava.util.logging.manager=org.jboss.logmanager.LogManager -XX:-UseBiasedLocking -XX:+UseStringDeduplication -XX:+UseNUMA -XX:+UseParallelGC -Djava.lang.Integer.IntegerCache.high=10000 -Dvertx.disableURIValidation=true -Dvertx.disableHttpHeadersValidation=true -Dvertx.disableH2c=true -Dvertx.disableWebsockets=true -Dvertx.flashPolicyHandler=false -Dvertx.threadChecks=false -Dvertx.disableContextTimings=true -Dhibernate.allow_update_outside_transaction=true -Dio.quarkus.vertx.core.runtime.context.VertxContextSafetyToggle.I_HAVE_CHECKED_EVERYTHING=true -Djboss.threads.eqe.statistics=false -Dmutiny.disableCallBackDecorators=true

Baseline2 Configuration : 

-server -Dquarkus.vertx.prefer-native-transport=true -XX:-StackTraceInThrowable -Dquarkus.http.accept-backlog=-1 -Dio.netty.buffer.checkBounds=false -Dio.netty.buffer.checkAccessible=false -Djava.util.logging.manager=org.jboss.logmanager.LogManager -XX:-UseBiasedLocking -XX:+UseStringDeduplication -XX:+UseNUMA -XX:+UseParallelGC -Djava.lang.Integer.IntegerCache.high=10000 -Dvertx.disableURIValidation=true -Dvertx.disableHttpHeadersValidation=true -Dvertx.disableH2c=true -Dvertx.disableWebsockets=true -Dvertx.flashPolicyHandler=false -Dvertx.threadChecks=false -Dvertx.disableContextTimings=true -Dhibernate.allow_update_outside_transaction=true -Dio.quarkus.vertx.core.runtime.context.VertxContextSafetyToggle.I_HAVE_CHECKED_EVERYTHING=true -Djboss.threads.eqe.statistics=false -Dmutiny.disableCallBackDecorators=true -Dquarkus.http.io-threads=4

Autotune Configuration: 

-server -XX:MaxRAMPercentage=70 -Dquarkus.thread-pool.core-threads=1 -Dquarkus.thread-pool.queue-size=7590 -Dquarkus.datasource.jdbc.min-size=8 -Dquarkus.datasource.jdbc.initial-size=8 -Dquarkus.datasource.jdbc.max-size=39 -Dquarkus.http.io-threads=4 -XX:FreqInlineSize=405 -XX:MaxInlineLevel=27 -XX:MinInliningThreshold=38 -XX:CompileThreshold=8740 -XX:CompileThresholdScaling=1 -XX:ConcGCThreads=1 -XX:InlineSmallCode=3257 -XX:LoopUnrollLimit=58 -XX:LoopUnrollMin=18 -XX:MinSurvivorRatio=7 -XX:NewRatio=1 -XX:TieredStopAtLevel=1 -XX:-TieredCompilation -XX:+AllowParallelDefineClass -XX:+AllowVectorizeOnDemand -XX:-AlwaysCompileLoopMethods -XX:-AlwaysPreTouch -XX:+AlwaysTenure -XX:+BackgroundCompilation -XX:+DoEscapeAnalysis -XX:+UseInlineCaches -XX:+UseLoopPredicate -XX:+UseStringDeduplication -XX:-UseSuperWord -XX:-UseTypeSpeculation -XX:-StackTraceInThrowable -XX:+UseParallelGC -Dio.netty.buffer.checkBounds=false -Dio.netty.buffer.checkAccessible=false

Throughput chart comparing baseline and Autotune config with TFB Quarkus v2.9.1.F:

Throughput

@Sanne @franz1981 @johnaohara @stalep @ddoliver @dinogun