search

DataDog / dd-trace-java

Datadog APM client for Java
https://docs.datadoghq.com/tracing/languages/java
Apache License 2.0
588 stars 290 forks source link

Bump jetty for testing to 9.4 #7776

Closed amarziali closed 1 month ago

amarziali commented 1 month ago

What Does This Do

Bumps jetty server version we use internally for TestHttpServer. We were sticking with 9.2 since it was the latest baselined with java 1.7 but since we dismissed java 1.7 support now we can upgrade to 9.4.x which is the latest baselined on 1.8

Motivation

Recent jetty server versions contains important fixes that can help to reduce flakiness (i.e. race conditions with servlet async handling)

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

pr-commenter[bot] commented 1 month ago

Kafka / producer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master andrea.marziali/bump-jetty-in-tests
git_commit_date 1729088733 1729160598
git_commit_sha 3f89f0321b9a5fdd6213bd85f6d54e94df482620 e46a0df4f4f4a6b407b231eed76f16e4954c9651
See matching parameters | |Baseline |Candidate | |---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |ci_job_date |1729161710 |1729161710 | |ci_job_id |675175497 |675175497 | |ci_pipeline_id|46803521 |46803521 | |cpu_model |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | |jdkVersion |11.0.21 |11.0.21 | |jmhVersion |1.36 |1.36 | |jvm |/usr/lib/jvm/java-11-openjdk-amd64/bin/java |/usr/lib/jvm/java-11-openjdk-amd64/bin/java | |jvmArgs |-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant|-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/producer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant| |vmName |OpenJDK 64-Bit Server VM |OpenJDK 64-Bit Server VM | |vmVersion |11.0.21+9-post-Ubuntu-0ubuntu122.04 |11.0.21+9-post-Ubuntu-0ubuntu122.04 |

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 2 metrics, 0 unstable metrics.

scenario Δ mean throughput
scenario:only-tracing-dsm-enabled-benchmarks/KafkaProduceBenchmark.benchProduce better
[+15501.325op/s; +20153.817op/s] or [+12.688%; +16.496%]
See unchanged results |scenario |Δ mean throughput| |--------------------------------------------------------------------------------|-----------------| |scenario:not-instrumented/KafkaProduceBenchmark.benchProduce |same | |scenario:only-tracing-dsm-disabled-benchmarks/KafkaProduceBenchmark.benchProduce|same |
pr-commenter[bot] commented 1 month ago

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master andrea.marziali/bump-jetty-in-tests
git_commit_date 1729113612 1729160598
git_commit_sha b53b6dc565 e46a0df4f4
release_version 1.41.0-SNAPSHOT~b53b6dc565 1.41.0-SNAPSHOT~e46a0df4f4
See matching parameters | |Baseline |Candidate | |---------------------|----------------------------------------------|----------------------------------------------| |application |insecure-bank |insecure-bank | |ci_job_date |1729163100 |1729163100 | |ci_job_id |675175493 |675175493 | |ci_pipeline_id|46803521 |46803521 | |cpu_model |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz| |module |Agent |Agent | |parent |None |None | |variant |iast |iast |

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 54 metrics, 9 unstable metrics.

Startup time reports for petclinic ```mermaid gantt title petclinic - global startup overhead: candidate=1.41.0-SNAPSHOT~e46a0df4f4, baseline=1.41.0-SNAPSHOT~b53b6dc565 dateFormat X axisFormat %s section tracing Agent [baseline] (1.076 s) : 0, 1076499 Total [baseline] (10.422 s) : 0, 10421780 Agent [candidate] (1.072 s) : 0, 1071957 Total [candidate] (10.442 s) : 0, 10442261 section appsec Agent [baseline] (1.212 s) : 0, 1211673 Total [baseline] (10.603 s) : 0, 10603241 Agent [candidate] (1.216 s) : 0, 1215584 Total [candidate] (10.642 s) : 0, 10641597 section iast Agent [baseline] (1.207 s) : 0, 1207462 Total [baseline] (10.9 s) : 0, 10899551 Agent [candidate] (1.212 s) : 0, 1211992 Total [candidate] (10.862 s) : 0, 10862418 section profiling Agent [baseline] (1.274 s) : 0, 1274477 Total [baseline] (10.713 s) : 0, 10712799 Agent [candidate] (1.284 s) : 0, 1283921 Total [candidate] (10.804 s) : 0, 10804497 ``` * **baseline** results |Module|Variant|Duration|Δ tracing| |---|---|---|---| |Agent|tracing|1.076 s |-| |Agent|appsec|1.212 s |135.174 ms (12.6%)| |Agent|iast|1.207 s |130.963 ms (12.2%)| |Agent|profiling|1.274 s |197.977 ms (18.4%)| |Total|tracing|10.422 s |-| |Total|appsec|10.603 s |181.461 ms (1.7%)| |Total|iast|10.9 s |477.771 ms (4.6%)| |Total|profiling|10.713 s |291.019 ms (2.8%)| * **candidate** results |Module|Variant|Duration|Δ tracing| |---|---|---|---| |Agent|tracing|1.072 s |-| |Agent|appsec|1.216 s |143.627 ms (13.4%)| |Agent|iast|1.212 s |140.034 ms (13.1%)| |Agent|profiling|1.284 s |211.964 ms (19.8%)| |Total|tracing|10.442 s |-| |Total|appsec|10.642 s |199.335 ms (1.9%)| |Total|iast|10.862 s |420.157 ms (4.0%)| |Total|profiling|10.804 s |362.236 ms (3.5%)| ```mermaid gantt title petclinic - break down per module: candidate=1.41.0-SNAPSHOT~e46a0df4f4, baseline=1.41.0-SNAPSHOT~b53b6dc565 dateFormat X axisFormat %s section tracing BytebuddyAgent [baseline] (686.267 ms) : 0, 686267 BytebuddyAgent [candidate] (682.496 ms) : 0, 682496 GlobalTracer [baseline] (314.066 ms) : 0, 314066 GlobalTracer [candidate] (313.506 ms) : 0, 313506 AppSec [baseline] (54.127 ms) : 0, 54127 AppSec [candidate] (54.08 ms) : 0, 54080 Remote Config [baseline] (672.051 µs) : 0, 672 Remote Config [candidate] (670.141 µs) : 0, 670 Telemetry [baseline] (7.598 ms) : 0, 7598 Telemetry [candidate] (7.506 ms) : 0, 7506 section appsec BytebuddyAgent [baseline] (702.731 ms) : 0, 702731 BytebuddyAgent [candidate] (706.654 ms) : 0, 706654 GlobalTracer [baseline] (311.469 ms) : 0, 311469 GlobalTracer [candidate] (312.793 ms) : 0, 312793 AppSec [baseline] (163.007 ms) : 0, 163007 AppSec [candidate] (162.798 ms) : 0, 162798 Remote Config [baseline] (644.67 µs) : 0, 645 Remote Config [candidate] (641.087 µs) : 0, 641 Telemetry [baseline] (10.004 ms) : 0, 10004 Telemetry [candidate] (7.433 ms) : 0, 7433 IAST [baseline] (20.181 ms) : 0, 20181 IAST [candidate] (22.839 ms) : 0, 22839 section iast BytebuddyAgent [baseline] (802.21 ms) : 0, 802210 BytebuddyAgent [candidate] (806.564 ms) : 0, 806564 GlobalTracer [baseline] (304.625 ms) : 0, 304625 GlobalTracer [candidate] (304.966 ms) : 0, 304966 AppSec [baseline] (58.241 ms) : 0, 58241 AppSec [candidate] (56.519 ms) : 0, 56519 Remote Config [baseline] (624.263 µs) : 0, 624 Remote Config [candidate] (683.639 µs) : 0, 684 Telemetry [baseline] (7.247 ms) : 0, 7247 Telemetry [candidate] (7.144 ms) : 0, 7144 IAST [baseline] (20.728 ms) : 0, 20728 IAST [candidate] (22.241 ms) : 0, 22241 section profiling ProfilingAgent [baseline] (96.784 ms) : 0, 96784 ProfilingAgent [candidate] (98.242 ms) : 0, 98242 BytebuddyAgent [baseline] (678.187 ms) : 0, 678187 BytebuddyAgent [candidate] (683.205 ms) : 0, 683205 GlobalTracer [baseline] (397.921 ms) : 0, 397921 GlobalTracer [candidate] (399.587 ms) : 0, 399587 AppSec [baseline] (54.67 ms) : 0, 54670 AppSec [candidate] (55.448 ms) : 0, 55448 Remote Config [baseline] (654.525 µs) : 0, 655 Remote Config [candidate] (680.244 µs) : 0, 680 Telemetry [baseline] (7.481 ms) : 0, 7481 Telemetry [candidate] (7.535 ms) : 0, 7535 Profiling [baseline] (96.808 ms) : 0, 96808 Profiling [candidate] (98.266 ms) : 0, 98266 ```
Startup time reports for insecure-bank ```mermaid gantt title insecure-bank - global startup overhead: candidate=1.41.0-SNAPSHOT~e46a0df4f4, baseline=1.41.0-SNAPSHOT~b53b6dc565 dateFormat X axisFormat %s section tracing Agent [baseline] (1.084 s) : 0, 1083783 Total [baseline] (8.657 s) : 0, 8656737 Agent [candidate] (1.074 s) : 0, 1073587 Total [candidate] (8.589 s) : 0, 8589054 section iast Agent [baseline] (1.214 s) : 0, 1214349 Total [baseline] (9.185 s) : 0, 9185345 Agent [candidate] (1.204 s) : 0, 1203976 Total [candidate] (9.124 s) : 0, 9123733 section iast_HARDCODED_SECRET_DISABLED Agent [baseline] (1.21 s) : 0, 1209975 Total [baseline] (9.139 s) : 0, 9138729 Agent [candidate] (1.218 s) : 0, 1217639 Total [candidate] (9.2 s) : 0, 9199652 section iast_TELEMETRY_OFF Agent [baseline] (1.197 s) : 0, 1197118 Total [baseline] (9.129 s) : 0, 9129419 Agent [candidate] (1.204 s) : 0, 1203929 Total [candidate] (9.152 s) : 0, 9151936 ``` * **baseline** results |Module|Variant|Duration|Δ tracing| |---|---|---|---| |Agent|tracing|1.084 s |-| |Agent|iast|1.214 s |130.566 ms (12.0%)| |Agent|iast_HARDCODED_SECRET_DISABLED|1.21 s |126.192 ms (11.6%)| |Agent|iast_TELEMETRY_OFF|1.197 s |113.335 ms (10.5%)| |Total|tracing|8.657 s |-| |Total|iast|9.185 s |528.608 ms (6.1%)| |Total|iast_HARDCODED_SECRET_DISABLED|9.139 s |481.993 ms (5.6%)| |Total|iast_TELEMETRY_OFF|9.129 s |472.683 ms (5.5%)| * **candidate** results |Module|Variant|Duration|Δ tracing| |---|---|---|---| |Agent|tracing|1.074 s |-| |Agent|iast|1.204 s |130.388 ms (12.1%)| |Agent|iast_HARDCODED_SECRET_DISABLED|1.218 s |144.052 ms (13.4%)| |Agent|iast_TELEMETRY_OFF|1.204 s |130.342 ms (12.1%)| |Total|tracing|8.589 s |-| |Total|iast|9.124 s |534.679 ms (6.2%)| |Total|iast_HARDCODED_SECRET_DISABLED|9.2 s |610.598 ms (7.1%)| |Total|iast_TELEMETRY_OFF|9.152 s |562.882 ms (6.6%)| ```mermaid gantt title insecure-bank - break down per module: candidate=1.41.0-SNAPSHOT~e46a0df4f4, baseline=1.41.0-SNAPSHOT~b53b6dc565 dateFormat X axisFormat %s section tracing BytebuddyAgent [baseline] (691.26 ms) : 0, 691260 BytebuddyAgent [candidate] (683.318 ms) : 0, 683318 GlobalTracer [baseline] (316.057 ms) : 0, 316057 GlobalTracer [candidate] (314.027 ms) : 0, 314027 AppSec [baseline] (54.351 ms) : 0, 54351 AppSec [candidate] (54.315 ms) : 0, 54315 Remote Config [baseline] (675.076 µs) : 0, 675 Remote Config [candidate] (673.013 µs) : 0, 673 Telemetry [baseline] (7.542 ms) : 0, 7542 Telemetry [candidate] (7.496 ms) : 0, 7496 section iast BytebuddyAgent [baseline] (808.354 ms) : 0, 808354 BytebuddyAgent [candidate] (800.585 ms) : 0, 800585 GlobalTracer [baseline] (305.181 ms) : 0, 305181 GlobalTracer [candidate] (303.229 ms) : 0, 303229 AppSec [baseline] (57.333 ms) : 0, 57333 AppSec [candidate] (57.162 ms) : 0, 57162 IAST [baseline] (21.742 ms) : 0, 21742 IAST [candidate] (21.391 ms) : 0, 21391 Remote Config [baseline] (623.281 µs) : 0, 623 Remote Config [candidate] (619.582 µs) : 0, 620 Telemetry [baseline] (7.235 ms) : 0, 7235 Telemetry [candidate] (7.166 ms) : 0, 7166 section iast_HARDCODED_SECRET_DISABLED BytebuddyAgent [baseline] (804.942 ms) : 0, 804942 BytebuddyAgent [candidate] (809.793 ms) : 0, 809793 GlobalTracer [baseline] (304.717 ms) : 0, 304717 GlobalTracer [candidate] (306.227 ms) : 0, 306227 AppSec [baseline] (57.364 ms) : 0, 57364 AppSec [candidate] (55.658 ms) : 0, 55658 IAST [baseline] (21.329 ms) : 0, 21329 IAST [candidate] (24.166 ms) : 0, 24166 Remote Config [baseline] (621.014 µs) : 0, 621 Remote Config [candidate] (619.499 µs) : 0, 619 Telemetry [baseline] (7.143 ms) : 0, 7143 Telemetry [candidate] (7.188 ms) : 0, 7188 section iast_TELEMETRY_OFF BytebuddyAgent [baseline] (795.149 ms) : 0, 795149 BytebuddyAgent [candidate] (799.694 ms) : 0, 799694 GlobalTracer [baseline] (302.395 ms) : 0, 302395 GlobalTracer [candidate] (304.116 ms) : 0, 304116 AppSec [baseline] (54.944 ms) : 0, 54944 AppSec [candidate] (54.214 ms) : 0, 54214 IAST [baseline] (23.258 ms) : 0, 23258 IAST [candidate] (24.343 ms) : 0, 24343 Remote Config [baseline] (611.277 µs) : 0, 611 Remote Config [candidate] (610.307 µs) : 0, 610 Telemetry [baseline] (6.999 ms) : 0, 6999 Telemetry [candidate] (7.079 ms) : 0, 7079 ```

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-10-17T10:35:07 2024-10-17T10:41:59
git_branch master andrea.marziali/bump-jetty-in-tests
git_commit_date 1729113612 1729160598
git_commit_sha b53b6dc565 e46a0df4f4
release_version 1.41.0-SNAPSHOT~b53b6dc565 1.41.0-SNAPSHOT~e46a0df4f4
start_time 2024-10-17T10:34:54 2024-10-17T10:41:45
See matching parameters | |Baseline |Candidate | |---------------------|----------------------------------------------|----------------------------------------------| |application |insecure-bank |insecure-bank | |ci_job_date |1729162064 |1729162064 | |ci_job_id |675175494 |675175494 | |ci_pipeline_id|46803521 |46803521 | |cpu_model |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz| |variant |iast |iast |

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for petclinic ```mermaid gantt title petclinic - request duration [CI 0.99] : candidate=1.41.0-SNAPSHOT~e46a0df4f4, baseline=1.41.0-SNAPSHOT~b53b6dc565 dateFormat X axisFormat %s section baseline no_agent (1.34 ms) : 1320, 1359 . : milestone, 1340, appsec (1.711 ms) : 1688, 1734 . : milestone, 1711, appsec_no_iast (1.734 ms) : 1711, 1758 . : milestone, 1734, iast (1.494 ms) : 1471, 1517 . : milestone, 1494, profiling (1.486 ms) : 1461, 1510 . : milestone, 1486, tracing (1.48 ms) : 1456, 1505 . : milestone, 1480, section candidate no_agent (1.323 ms) : 1304, 1342 . : milestone, 1323, appsec (1.734 ms) : 1711, 1757 . : milestone, 1734, appsec_no_iast (1.718 ms) : 1693, 1742 . : milestone, 1718, iast (1.479 ms) : 1457, 1502 . : milestone, 1479, profiling (1.473 ms) : 1450, 1497 . : milestone, 1473, tracing (1.465 ms) : 1441, 1490 . : milestone, 1465, ``` * **baseline** results |Variant|Request duration [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|1.34 ms [1.32 ms, 1.359 ms]|-| |appsec|1.711 ms [1.688 ms, 1.734 ms]|371.12 µs (27.7%)| |appsec_no_iast|1.734 ms [1.711 ms, 1.758 ms]|394.759 µs (29.5%)| |iast|1.494 ms [1.471 ms, 1.517 ms]|154.134 µs (11.5%)| |profiling|1.486 ms [1.461 ms, 1.51 ms]|145.899 µs (10.9%)| |tracing|1.48 ms [1.456 ms, 1.505 ms]|140.693 µs (10.5%)| * **candidate** results |Variant|Request duration [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|1.323 ms [1.304 ms, 1.342 ms]|-| |appsec|1.734 ms [1.711 ms, 1.757 ms]|410.388 µs (31.0%)| |appsec_no_iast|1.718 ms [1.693 ms, 1.742 ms]|394.291 µs (29.8%)| |iast|1.479 ms [1.457 ms, 1.502 ms]|155.893 µs (11.8%)| |profiling|1.473 ms [1.45 ms, 1.497 ms]|149.973 µs (11.3%)| |tracing|1.465 ms [1.441 ms, 1.49 ms]|142.086 µs (10.7%)|
Request duration reports for insecure-bank ```mermaid gantt title insecure-bank - request duration [CI 0.99] : candidate=1.41.0-SNAPSHOT~e46a0df4f4, baseline=1.41.0-SNAPSHOT~b53b6dc565 dateFormat X axisFormat %s section baseline no_agent (375.558 µs) : 355, 396 . : milestone, 376, iast (479.199 µs) : 458, 500 . : milestone, 479, iast_FULL (558.683 µs) : 537, 580 . : milestone, 559, iast_GLOBAL (520.721 µs) : 498, 543 . : milestone, 521, iast_HARDCODED_SECRET_DISABLED (483.566 µs) : 462, 506 . : milestone, 484, iast_INACTIVE (444.715 µs) : 424, 465 . : milestone, 445, iast_TELEMETRY_OFF (479.054 µs) : 457, 501 . : milestone, 479, tracing (445.596 µs) : 425, 466 . : milestone, 446, section candidate no_agent (370.955 µs) : 352, 390 . : milestone, 371, iast (487.284 µs) : 466, 509 . : milestone, 487, iast_FULL (558.739 µs) : 537, 581 . : milestone, 559, iast_GLOBAL (506.587 µs) : 485, 528 . : milestone, 507, iast_HARDCODED_SECRET_DISABLED (483.619 µs) : 462, 505 . : milestone, 484, iast_INACTIVE (446.012 µs) : 425, 467 . : milestone, 446, iast_TELEMETRY_OFF (473.45 µs) : 452, 495 . : milestone, 473, tracing (438.273 µs) : 418, 458 . : milestone, 438, ``` * **baseline** results |Variant|Request duration [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|375.558 µs [354.974 µs, 396.142 µs]|-| |iast|479.199 µs [458.18 µs, 500.218 µs]|103.642 µs (27.6%)| |iast_FULL|558.683 µs [537.151 µs, 580.215 µs]|183.125 µs (48.8%)| |iast_GLOBAL|520.721 µs [498.09 µs, 543.351 µs]|145.163 µs (38.7%)| |iast_HARDCODED_SECRET_DISABLED|483.566 µs [461.552 µs, 505.58 µs]|108.008 µs (28.8%)| |iast_INACTIVE|444.715 µs [424.206 µs, 465.223 µs]|69.157 µs (18.4%)| |iast_TELEMETRY_OFF|479.054 µs [456.948 µs, 501.16 µs]|103.496 µs (27.6%)| |tracing|445.596 µs [424.942 µs, 466.25 µs]|70.038 µs (18.6%)| * **candidate** results |Variant|Request duration [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|370.955 µs [351.523 µs, 390.388 µs]|-| |iast|487.284 µs [465.887 µs, 508.68 µs]|116.328 µs (31.4%)| |iast_FULL|558.739 µs [536.882 µs, 580.596 µs]|187.784 µs (50.6%)| |iast_GLOBAL|506.587 µs [485.495 µs, 527.679 µs]|135.631 µs (36.6%)| |iast_HARDCODED_SECRET_DISABLED|483.619 µs [462.38 µs, 504.858 µs]|112.663 µs (30.4%)| |iast_INACTIVE|446.012 µs [424.616 µs, 467.408 µs]|75.056 µs (20.2%)| |iast_TELEMETRY_OFF|473.45 µs [452.312 µs, 494.587 µs]|102.494 µs (27.6%)| |tracing|438.273 µs [418.076 µs, 458.47 µs]|67.317 µs (18.1%)|

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master andrea.marziali/bump-jetty-in-tests
git_commit_date 1729113612 1729160598
git_commit_sha b53b6dc565 e46a0df4f4
release_version 1.41.0-SNAPSHOT~b53b6dc565 1.41.0-SNAPSHOT~e46a0df4f4
See matching parameters | |Baseline |Candidate | |---------------------|----------------------------------------------|----------------------------------------------| |application |biojava |biojava | |ci_job_date |1729162594 |1729162594 | |ci_job_id |675175495 |675175495 | |ci_pipeline_id|46803521 |46803521 | |cpu_model |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz| |variant |appsec |appsec |

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava ```mermaid gantt title biojava - execution time [CI 0.99] : candidate=1.41.0-SNAPSHOT~e46a0df4f4, baseline=1.41.0-SNAPSHOT~b53b6dc565 dateFormat X axisFormat %s section baseline no_agent (14.976 s) : 14976000, 14976000 . : milestone, 14976000, appsec (15.264 s) : 15264000, 15264000 . : milestone, 15264000, iast (18.976 s) : 18976000, 18976000 . : milestone, 18976000, iast_GLOBAL (17.857 s) : 17857000, 17857000 . : milestone, 17857000, profiling (15.139 s) : 15139000, 15139000 . : milestone, 15139000, tracing (15.124 s) : 15124000, 15124000 . : milestone, 15124000, section candidate no_agent (15.717 s) : 15717000, 15717000 . : milestone, 15717000, appsec (15.325 s) : 15325000, 15325000 . : milestone, 15325000, iast (19.058 s) : 19058000, 19058000 . : milestone, 19058000, iast_GLOBAL (18.102 s) : 18102000, 18102000 . : milestone, 18102000, profiling (15.426 s) : 15426000, 15426000 . : milestone, 15426000, tracing (14.864 s) : 14864000, 14864000 . : milestone, 14864000, ``` * **baseline** results |Variant|Execution Time [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|14.976 s [14.976 s, 14.976 s]|-| |appsec|15.264 s [15.264 s, 15.264 s]|288.0 ms (1.9%)| |iast|18.976 s [18.976 s, 18.976 s]|4.0 s (26.7%)| |iast_GLOBAL|17.857 s [17.857 s, 17.857 s]|2.881 s (19.2%)| |profiling|15.139 s [15.139 s, 15.139 s]|163.0 ms (1.1%)| |tracing|15.124 s [15.124 s, 15.124 s]|148.0 ms (1.0%)| * **candidate** results |Variant|Execution Time [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|15.717 s [15.717 s, 15.717 s]|-| |appsec|15.325 s [15.325 s, 15.325 s]|-392.0 ms (-2.5%)| |iast|19.058 s [19.058 s, 19.058 s]|3.341 s (21.3%)| |iast_GLOBAL|18.102 s [18.102 s, 18.102 s]|2.385 s (15.2%)| |profiling|15.426 s [15.426 s, 15.426 s]|-291.0 ms (-1.9%)| |tracing|14.864 s [14.864 s, 14.864 s]|-853.0 ms (-5.4%)|
Execution time for tomcat ```mermaid gantt title tomcat - execution time [CI 0.99] : candidate=1.41.0-SNAPSHOT~e46a0df4f4, baseline=1.41.0-SNAPSHOT~b53b6dc565 dateFormat X axisFormat %s section baseline no_agent (1.467 ms) : 1455, 1478 . : milestone, 1467, appsec (2.329 ms) : 2288, 2370 . : milestone, 2329, iast (2.073 ms) : 2021, 2125 . : milestone, 2073, iast_GLOBAL (2.123 ms) : 2070, 2176 . : milestone, 2123, profiling (1.936 ms) : 1895, 1977 . : milestone, 1936, tracing (1.912 ms) : 1873, 1951 . : milestone, 1912, section candidate no_agent (1.476 ms) : 1464, 1487 . : milestone, 1476, appsec (2.343 ms) : 2302, 2384 . : milestone, 2343, iast (2.069 ms) : 2017, 2120 . : milestone, 2069, iast_GLOBAL (2.124 ms) : 2071, 2176 . : milestone, 2124, profiling (1.943 ms) : 1902, 1984 . : milestone, 1943, tracing (1.917 ms) : 1878, 1957 . : milestone, 1917, ``` * **baseline** results |Variant|Execution Time [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|1.467 ms [1.455 ms, 1.478 ms]|-| |appsec|2.329 ms [2.288 ms, 2.37 ms]|861.777 µs (58.7%)| |iast|2.073 ms [2.021 ms, 2.125 ms]|605.889 µs (41.3%)| |iast_GLOBAL|2.123 ms [2.07 ms, 2.176 ms]|656.221 µs (44.7%)| |profiling|1.936 ms [1.895 ms, 1.977 ms]|469.043 µs (32.0%)| |tracing|1.912 ms [1.873 ms, 1.951 ms]|445.184 µs (30.3%)| * **candidate** results |Variant|Execution Time [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|1.476 ms [1.464 ms, 1.487 ms]|-| |appsec|2.343 ms [2.302 ms, 2.384 ms]|867.202 µs (58.8%)| |iast|2.069 ms [2.017 ms, 2.12 ms]|592.823 µs (40.2%)| |iast_GLOBAL|2.124 ms [2.071 ms, 2.176 ms]|647.925 µs (43.9%)| |profiling|1.943 ms [1.902 ms, 1.984 ms]|467.292 µs (31.7%)| |tracing|1.917 ms [1.878 ms, 1.957 ms]|441.642 µs (29.9%)|
pr-commenter[bot] commented 1 month ago

Kafka / consumer-benchmark

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master andrea.marziali/bump-jetty-in-tests
git_commit_date 1729088733 1729160598
git_commit_sha 3f89f0321b9a5fdd6213bd85f6d54e94df482620 e46a0df4f4f4a6b407b231eed76f16e4954c9651
See matching parameters | |Baseline |Candidate | |---------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |ci_job_date |1729161750 |1729161750 | |ci_job_id |675175498 |675175498 | |ci_pipeline_id|46803521 |46803521 | |cpu_model |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | |jdkVersion |11.0.21 |11.0.21 | |jmhVersion |1.36 |1.36 | |jvm |/usr/lib/jvm/java-11-openjdk-amd64/bin/java |/usr/lib/jvm/java-11-openjdk-amd64/bin/java | |jvmArgs |-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant|-Dfile.encoding=UTF-8 -Djava.io.tmpdir=/go/src/github.com/DataDog/apm-reliability/dd-trace-java/platform/src/consumer-benchmark/build/tmp/jmh -Duser.country=US -Duser.language=en -Duser.variant| |vmName |OpenJDK 64-Bit Server VM |OpenJDK 64-Bit Server VM | |vmVersion |11.0.21+9-post-Ubuntu-0ubuntu122.04 |11.0.21+9-post-Ubuntu-0ubuntu122.04 |

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 3 metrics, 0 unstable metrics.

See unchanged results |scenario |Δ mean throughput | |---------------------------------------------------------------------------------|--------------------------------------------------------------------| |scenario:not-instrumented/KafkaConsumerBenchmark.benchConsume |same | |scenario:only-tracing-dsm-disabled-benchmarks/KafkaConsumerBenchmark.benchConsume|same | |scenario:only-tracing-dsm-enabled-benchmarks/KafkaConsumerBenchmark.benchConsume |unsure
[-6391.051op/s; -105.942op/s] or [-3.583%; -0.059%]|
pr-commenter[bot] commented 1 month ago

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1729161238 1729161614
end_time 2024-10-17T10:35:13 2024-10-17T10:41:28
git_branch master andrea.marziali/bump-jetty-in-tests
git_commit_sha b53b6dc565 e46a0df4f4f4a6b407b231eed76f16e4954c9651
start_time 2024-10-17T10:33:59 2024-10-17T10:40:15
See matching parameters | |Baseline |Candidate | |----------------------|----------------------------------------------|----------------------------------------------| |ci_job_id |675175499 |675175499 | |ci_pipeline_id |46803521 |46803521 | |cpu_model |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz| |git_commit_date|1729160598 |1729160598 |

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 6 unstable metrics.

See unchanged results |scenario |Δ mean agg_http_req_duration_min |Δ mean agg_http_req_duration_p50 |Δ mean agg_http_req_duration_p75 |Δ mean agg_http_req_duration_p99 |Δ mean throughput | |----------------|---------------------------------------------------------------|----------------------------------------------------------------|-----------------------------------------------------------------|-----------------------------------------------------------------|---------------------------------------------------------------------| |scenario:noprobe|unstable
[-24.788µs; +16.405µs] or [-9.456%; +6.258%]|unstable
[-38.153µs; +24.671µs] or [-12.663%; +8.189%]|unstable
[-51.811µs; +34.581µs] or [-16.398%; +10.944%]|unstable
[-93.881µs; +90.907µs] or [-15.340%; +14.854%]|same | |scenario:basic |same |same |same |unstable
[-41.237µs; +46.510µs] or [-7.545%; +8.509%] |unstable
[-140.754op/s; +140.754op/s] or [-5.349%; +5.349%]| |scenario:loop |unsure
[-35.881µs; -21.413µs] or [-0.353%; -0.211%] |same |same |same |same |
Request duration reports for reports ```mermaid gantt title reports - request duration [CI 0.99] : candidate=None, baseline=None dateFormat X axisFormat %s section baseline noprobe (301.284 µs) : 266, 337 . : milestone, 301, basic (297.678 µs) : 288, 307 . : milestone, 298, loop (10.329 ms) : 10300, 10358 . : milestone, 10329, section candidate noprobe (294.542 µs) : 271, 318 . : milestone, 295, basic (294.768 µs) : 285, 304 . : milestone, 295, loop (10.308 ms) : 10279, 10336 . : milestone, 10308, ``` * **baseline** results |Scenario|Request median duration [CI 0.99]| |---|---| |noprobe|301.284 µs [266.054 µs, 336.514 µs]| |basic|297.678 µs [288.254 µs, 307.102 µs]| |loop|10.329 ms [10.3 ms, 10.358 ms]| * **candidate** results |Scenario|Request median duration [CI 0.99]| |---|---| |noprobe|294.542 µs [270.989 µs, 318.096 µs]| |basic|294.768 µs [285.084 µs, 304.452 µs]| |loop|10.308 ms [10.279 ms, 10.336 ms]|