search

DataDog / dd-trace-java

Datadog APM client for Java
https://docs.datadoghq.com/tracing/languages/java
Apache License 2.0
572 stars 284 forks source link

Retry JettyServlet3ServeFromAsyncTimeout on socket timeout #7625

Closed smola closed 2 weeks ago

smola commented 2 weeks ago

What Does This Do

Retry tests in JettyServlet3ServeFromAsyncTimeout when there's a SocketTimeoutException. This needs to be done with SETUP_FEATURE_CLEANUP to ensure that cleanup is run, and leftover spans do not leak between iterations.

Motivation

This test suite is, by far, the most flaky in our test suites. However, being in a very widely used code path, I'm reluctant to simply mark as @Flaky and hide any regression here.

Additional Notes

Contributor Checklist

pr-commenter[bot] commented 2 weeks ago

Benchmarks

Startup

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-09-16T10:09:40 2024-09-16T10:16:33
git_branch master smola/jetty-async-timeout-retry-test
git_commit_date 1726476601 1726480836
git_commit_sha 7108ba0fab bbcdc4406b
release_version 1.40.0-SNAPSHOT~7108ba0fab 1.40.0-SNAPSHOT~bbcdc4406b
start_time 2024-09-16T10:09:26 2024-09-16T10:16:20
See matching parameters | |Baseline |Candidate | |---------------------|----------------------------------------------|----------------------------------------------| |application |insecure-bank |insecure-bank | |ci_job_date |1726482142 |1726482142 | |ci_job_id |640231038 |640231038 | |ci_pipeline_id|44396930 |44396930 | |cpu_model |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz| |variant |iast |iast |

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for insecure-bank ```mermaid gantt title insecure-bank - request duration [CI 0.99] : candidate=1.40.0-SNAPSHOT~bbcdc4406b, baseline=1.40.0-SNAPSHOT~7108ba0fab dateFormat X axisFormat %s section baseline no_agent (377.853 µs) : 358, 397 . : milestone, 378, iast (496.727 µs) : 475, 518 . : milestone, 497, iast_FULL (561.569 µs) : 540, 583 . : milestone, 562, iast_GLOBAL (520.765 µs) : 499, 543 . : milestone, 521, iast_HARDCODED_SECRET_DISABLED (489.767 µs) : 469, 511 . : milestone, 490, iast_INACTIVE (455.099 µs) : 434, 476 . : milestone, 455, iast_TELEMETRY_OFF (482.747 µs) : 460, 506 . : milestone, 483, tracing (449.337 µs) : 429, 470 . : milestone, 449, section candidate no_agent (379.58 µs) : 359, 400 . : milestone, 380, iast (490.766 µs) : 470, 512 . : milestone, 491, iast_FULL (565.76 µs) : 544, 588 . : milestone, 566, iast_GLOBAL (509.271 µs) : 488, 530 . : milestone, 509, iast_HARDCODED_SECRET_DISABLED (498.508 µs) : 477, 520 . : milestone, 499, iast_INACTIVE (457.244 µs) : 436, 479 . : milestone, 457, iast_TELEMETRY_OFF (491.495 µs) : 469, 514 . : milestone, 491, tracing (448.023 µs) : 428, 468 . : milestone, 448, ``` * **baseline** results |Variant|Request duration [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|377.853 µs [358.307 µs, 397.399 µs]|-| |iast|496.727 µs [475.1 µs, 518.354 µs]|118.874 µs (31.5%)| |iast_FULL|561.569 µs [540.231 µs, 582.907 µs]|183.715 µs (48.6%)| |iast_GLOBAL|520.765 µs [498.68 µs, 542.85 µs]|142.912 µs (37.8%)| |iast_HARDCODED_SECRET_DISABLED|489.767 µs [468.801 µs, 510.733 µs]|111.914 µs (29.6%)| |iast_INACTIVE|455.099 µs [434.057 µs, 476.141 µs]|77.246 µs (20.4%)| |iast_TELEMETRY_OFF|482.747 µs [459.715 µs, 505.779 µs]|104.894 µs (27.8%)| |tracing|449.337 µs [428.936 µs, 469.738 µs]|71.484 µs (18.9%)| * **candidate** results |Variant|Request duration [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|379.58 µs [359.193 µs, 399.967 µs]|-| |iast|490.766 µs [469.554 µs, 511.978 µs]|111.186 µs (29.3%)| |iast_FULL|565.76 µs [543.916 µs, 587.603 µs]|186.18 µs (49.0%)| |iast_GLOBAL|509.271 µs [488.107 µs, 530.435 µs]|129.691 µs (34.2%)| |iast_HARDCODED_SECRET_DISABLED|498.508 µs [477.072 µs, 519.944 µs]|118.928 µs (31.3%)| |iast_INACTIVE|457.244 µs [435.947 µs, 478.541 µs]|77.664 µs (20.5%)| |iast_TELEMETRY_OFF|491.495 µs [468.712 µs, 514.277 µs]|111.915 µs (29.5%)| |tracing|448.023 µs [427.818 µs, 468.227 µs]|68.443 µs (18.0%)|
Request duration reports for petclinic ```mermaid gantt title petclinic - request duration [CI 0.99] : candidate=1.40.0-SNAPSHOT~bbcdc4406b, baseline=1.40.0-SNAPSHOT~7108ba0fab dateFormat X axisFormat %s section baseline no_agent (1.342 ms) : 1322, 1362 . : milestone, 1342, appsec (1.717 ms) : 1694, 1740 . : milestone, 1717, appsec_no_iast (1.739 ms) : 1715, 1763 . : milestone, 1739, iast (1.485 ms) : 1462, 1508 . : milestone, 1485, profiling (1.493 ms) : 1469, 1517 . : milestone, 1493, tracing (1.493 ms) : 1469, 1517 . : milestone, 1493, section candidate no_agent (1.362 ms) : 1342, 1381 . : milestone, 1362, appsec (1.731 ms) : 1709, 1754 . : milestone, 1731, appsec_no_iast (1.738 ms) : 1714, 1762 . : milestone, 1738, iast (1.477 ms) : 1454, 1499 . : milestone, 1477, profiling (1.508 ms) : 1484, 1533 . : milestone, 1508, tracing (1.466 ms) : 1440, 1492 . : milestone, 1466, ``` * **baseline** results |Variant|Request duration [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|1.342 ms [1.322 ms, 1.362 ms]|-| |appsec|1.717 ms [1.694 ms, 1.74 ms]|374.972 µs (27.9%)| |appsec_no_iast|1.739 ms [1.715 ms, 1.763 ms]|397.143 µs (29.6%)| |iast|1.485 ms [1.462 ms, 1.508 ms]|142.969 µs (10.7%)| |profiling|1.493 ms [1.469 ms, 1.517 ms]|150.943 µs (11.2%)| |tracing|1.493 ms [1.469 ms, 1.517 ms]|150.955 µs (11.2%)| * **candidate** results |Variant|Request duration [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|1.362 ms [1.342 ms, 1.381 ms]|-| |appsec|1.731 ms [1.709 ms, 1.754 ms]|369.792 µs (27.2%)| |appsec_no_iast|1.738 ms [1.714 ms, 1.762 ms]|376.324 µs (27.6%)| |iast|1.477 ms [1.454 ms, 1.499 ms]|114.918 µs (8.4%)| |profiling|1.508 ms [1.484 ms, 1.533 ms]|146.66 µs (10.8%)| |tracing|1.466 ms [1.44 ms, 1.492 ms]|104.472 µs (7.7%)|

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master smola/jetty-async-timeout-retry-test
git_commit_date 1726476601 1726480836
git_commit_sha 7108ba0fab bbcdc4406b
release_version 1.40.0-SNAPSHOT~7108ba0fab 1.40.0-SNAPSHOT~bbcdc4406b
See matching parameters | |Baseline |Candidate | |---------------------|----------------------------------------------|----------------------------------------------| |application |biojava |biojava | |ci_job_date |1726482671 |1726482671 | |ci_job_id |640231039 |640231039 | |ci_pipeline_id|44396930 |44396930 | |cpu_model |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz| |variant |appsec |appsec |

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat ```mermaid gantt title tomcat - execution time [CI 0.99] : candidate=1.40.0-SNAPSHOT~bbcdc4406b, baseline=1.40.0-SNAPSHOT~7108ba0fab dateFormat X axisFormat %s section baseline no_agent (1.474 ms) : 1463, 1486 . : milestone, 1474, appsec (2.323 ms) : 2281, 2365 . : milestone, 2323, iast (2.07 ms) : 2019, 2120 . : milestone, 2070, iast_GLOBAL (2.096 ms) : 2045, 2147 . : milestone, 2096, profiling (1.929 ms) : 1888, 1969 . : milestone, 1929, tracing (1.898 ms) : 1860, 1937 . : milestone, 1898, section candidate no_agent (1.468 ms) : 1456, 1480 . : milestone, 1468, appsec (2.294 ms) : 2254, 2335 . : milestone, 2294, iast (2.054 ms) : 2004, 2104 . : milestone, 2054, iast_GLOBAL (2.104 ms) : 2052, 2155 . : milestone, 2104, profiling (1.923 ms) : 1881, 1964 . : milestone, 1923, tracing (1.895 ms) : 1857, 1934 . : milestone, 1895, ``` * **baseline** results |Variant|Execution Time [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|1.474 ms [1.463 ms, 1.486 ms]|-| |appsec|2.323 ms [2.281 ms, 2.365 ms]|848.913 µs (57.6%)| |iast|2.07 ms [2.019 ms, 2.12 ms]|595.537 µs (40.4%)| |iast_GLOBAL|2.096 ms [2.045 ms, 2.147 ms]|621.629 µs (42.2%)| |profiling|1.929 ms [1.888 ms, 1.969 ms]|454.195 µs (30.8%)| |tracing|1.898 ms [1.86 ms, 1.937 ms]|423.894 µs (28.7%)| * **candidate** results |Variant|Execution Time [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|1.468 ms [1.456 ms, 1.48 ms]|-| |appsec|2.294 ms [2.254 ms, 2.335 ms]|826.326 µs (56.3%)| |iast|2.054 ms [2.004 ms, 2.104 ms]|586.01 µs (39.9%)| |iast_GLOBAL|2.104 ms [2.052 ms, 2.155 ms]|635.514 µs (43.3%)| |profiling|1.923 ms [1.881 ms, 1.964 ms]|454.634 µs (31.0%)| |tracing|1.895 ms [1.857 ms, 1.934 ms]|427.432 µs (29.1%)|
Execution time for biojava ```mermaid gantt title biojava - execution time [CI 0.99] : candidate=1.40.0-SNAPSHOT~bbcdc4406b, baseline=1.40.0-SNAPSHOT~7108ba0fab dateFormat X axisFormat %s section baseline no_agent (14.908 s) : 14908000, 14908000 . : milestone, 14908000, appsec (15.32 s) : 15320000, 15320000 . : milestone, 15320000, iast (18.881 s) : 18881000, 18881000 . : milestone, 18881000, iast_GLOBAL (18.221 s) : 18221000, 18221000 . : milestone, 18221000, profiling (14.975 s) : 14975000, 14975000 . : milestone, 14975000, tracing (15.0 s) : 15000000, 15000000 . : milestone, 15000000, section candidate no_agent (15.09 s) : 15090000, 15090000 . : milestone, 15090000, appsec (15.128 s) : 15128000, 15128000 . : milestone, 15128000, iast (19.093 s) : 19093000, 19093000 . : milestone, 19093000, iast_GLOBAL (18.454 s) : 18454000, 18454000 . : milestone, 18454000, profiling (15.767 s) : 15767000, 15767000 . : milestone, 15767000, tracing (14.805 s) : 14805000, 14805000 . : milestone, 14805000, ``` * **baseline** results |Variant|Execution Time [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|14.908 s [14.908 s, 14.908 s]|-| |appsec|15.32 s [15.32 s, 15.32 s]|412.0 ms (2.8%)| |iast|18.881 s [18.881 s, 18.881 s]|3.973 s (26.7%)| |iast_GLOBAL|18.221 s [18.221 s, 18.221 s]|3.313 s (22.2%)| |profiling|14.975 s [14.975 s, 14.975 s]|67.0 ms (0.4%)| |tracing|15.0 s [15.0 s, 15.0 s]|92.0 ms (0.6%)| * **candidate** results |Variant|Execution Time [CI 0.99]|Δ no_agent| |---|---|---| |no_agent|15.09 s [15.09 s, 15.09 s]|-| |appsec|15.128 s [15.128 s, 15.128 s]|38.0 ms (0.3%)| |iast|19.093 s [19.093 s, 19.093 s]|4.003 s (26.5%)| |iast_GLOBAL|18.454 s [18.454 s, 18.454 s]|3.364 s (22.3%)| |profiling|15.767 s [15.767 s, 15.767 s]|677.0 ms (4.5%)| |tracing|14.805 s [14.805 s, 14.805 s]|-285.0 ms (-1.9%)|