pr-commenter[bot]
commented
1 week ago Benchmarks
Comparison
Benchmark execution time: 2024-11-14 20:55:43
Comparing candidate commit 3cd888a3 in PR branch dsn/reduce-fuzz-time with baseline commit 0ded7063 in branch main.
Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 2 unstable metrics.
Candidate
Candidate benchmark details
### Group 1 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |----------------------------------------------------------------------------|--------------|--------|------------------|------------------|--------|--------|--------|--------|--------------------|--------|--------|-----|-------|----|-----------| |benching deserializing traces from msgpack to their internal representation |execution_time|59.195ms|59.417ms ± 0.188ms|59.376ms ± 0.045ms|59.425ms|59.592ms|60.435ms|60.683ms|2.20% |4.517 |23.356 |0.32%|0.013ms|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |----------------------------------------------------------------------------|--------------|------------------------------------------|-------------------|------------------------|---------------| |benching deserializing traces from msgpack to their internal representation |execution_time|[59.391ms; 59.444ms] or [-0.044%; +0.044%]|None |None |None | ### Group 2 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |----------------------------------------------------------|--------------|-----------------|----------------------------------|----------------------------------|-----------------|-----------------|-----------------|-----------------|--------------------|--------|--------|-----|-------------|----|-----------| |credit_card/is_card_number/ |execution_time|4.599µs |4.613µs ± 0.008µs |4.613µs ± 0.006µs |4.618µs |4.628µs |4.631µs |4.632µs |0.42% |0.296 |-0.633 |0.18%|0.001µs |1 |200 | |credit_card/is_card_number/ |throughput |215869245.938op/s|216793325.728op/s ± 391812.376op/s|216777845.685op/s ± 268454.385op/s|217121958.686op/s|217366510.920op/s|217394648.720op/s|217432099.835op/s|0.30% |-0.289 |-0.640 |0.18%|27705.319op/s|1 |200 | |credit_card/is_card_number/ 3782-8224-6310-005 |execution_time|93.004µs |93.508µs ± 0.366µs |93.445µs ± 0.104µs |93.559µs |94.114µs |94.340µs |97.522µs |4.36% |7.044 |71.171 |0.39%|0.026µs |1 |200 | |credit_card/is_card_number/ 3782-8224-6310-005 |throughput |10254106.338op/s |10694430.835op/s ± 40745.050op/s |10701461.826op/s ± 11903.756op/s |10712890.286op/s |10719370.824op/s |10732092.720op/s |10752178.170op/s |0.47% |-6.774 |66.976 |0.38%|2881.110op/s |1 |200 | |credit_card/is_card_number/ 378282246310005 |execution_time|85.193µs |85.395µs ± 0.366µs |85.316µs ± 0.026µs |85.343µs |85.856µs |86.352µs |89.417µs |4.81% |7.759 |75.237 |0.43%|0.026µs |1 |200 | |credit_card/is_card_number/ 378282246310005 |throughput |11183583.258op/s |11710430.880op/s ± 48564.923op/s |11721063.909op/s ± 3529.369op/s |11724566.193op/s |11728670.041op/s |11732853.443op/s |11738058.267op/s |0.14% |-7.547 |71.609 |0.41%|3434.059op/s |1 |200 | |credit_card/is_card_number/37828224631 |execution_time|4.593µs |4.612µs ± 0.009µs |4.611µs ± 0.007µs |4.618µs |4.628µs |4.632µs |4.634µs |0.50% |0.378 |-0.588 |0.19%|0.001µs |1 |200 | |credit_card/is_card_number/37828224631 |throughput |215786339.062op/s|216823544.183op/s ± 413075.633op/s|216860932.498op/s ± 320689.473op/s|217176392.731op/s|217397051.036op/s|217416364.747op/s|217723780.762op/s|0.40% |-0.371 |-0.596 |0.19%|29208.858op/s|1 |200 | |credit_card/is_card_number/378282246310005 |execution_time|81.613µs |81.968µs ± 0.169µs |81.966µs ± 0.125µs |82.089µs |82.218µs |82.331µs |82.836µs |1.06% |0.879 |2.980 |0.21%|0.012µs |1 |200 | |credit_card/is_card_number/378282246310005 |throughput |12072084.196op/s |12199932.925op/s ± 25077.190op/s |12200112.248op/s ± 18635.837op/s |12219906.849op/s |12235083.154op/s |12248230.916op/s |12252963.434op/s |0.43% |-0.854 |2.857 |0.21%|1773.225op/s |1 |200 | |credit_card/is_card_number/37828224631000521389798 |execution_time|58.861µs |58.890µs ± 0.022µs |58.888µs ± 0.015µs |58.902µs |58.927µs |58.949µs |59.021µs |0.23% |1.512 |4.943 |0.04%|0.002µs |1 |200 | |credit_card/is_card_number/37828224631000521389798 |throughput |16943086.478op/s |16980763.115op/s ± 6457.050op/s |16981399.424op/s ± 4428.147op/s |16986082.878op/s |16987866.671op/s |16988779.930op/s |16989055.650op/s |0.05% |-1.506 |4.906 |0.04%|456.582op/s |1 |200 | |credit_card/is_card_number/x371413321323331 |execution_time|6.819µs |6.834µs ± 0.026µs |6.823µs ± 0.003µs |6.826µs |6.900µs |6.906µs |6.980µs |2.30% |2.502 |6.902 |0.37%|0.002µs |1 |200 | |credit_card/is_card_number/x371413321323331 |throughput |143266658.030op/s|146332092.561op/s ± 541262.046op/s|146559674.431op/s ± 57950.241op/s |146615465.183op/s|146643360.296op/s|146653136.647op/s|146658947.775op/s|0.07% |-2.474 |6.666 |0.37%|38273.006op/s|1 |200 | |credit_card/is_card_number_no_luhn/ |execution_time|4.593µs |4.612µs ± 0.008µs |4.612µs ± 0.005µs |4.617µs |4.628µs |4.630µs |4.631µs |0.41% |0.320 |-0.412 |0.17%|0.001µs |1 |200 | |credit_card/is_card_number_no_luhn/ |throughput |215952895.387op/s|216815837.772op/s ± 373166.552op/s|216831210.336op/s ± 253769.923op/s|217089269.279op/s|217379923.826op/s|217421885.579op/s|217706581.112op/s|0.40% |-0.313 |-0.418 |0.17%|26386.860op/s|1 |200 | |credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 |execution_time|73.413µs |73.695µs ± 0.126µs |73.714µs ± 0.070µs |73.766µs |73.838µs |74.009µs |74.470µs |1.03% |0.940 |6.498 |0.17%|0.009µs |1 |200 | |credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 |throughput |13428231.435op/s |13569411.744op/s ± 23084.044op/s |13565876.395op/s ± 12789.726op/s |13584174.438op/s |13610456.710op/s |13617482.539op/s |13621636.608op/s |0.41% |-0.902 |6.264 |0.17%|1632.288op/s |1 |200 | |credit_card/is_card_number_no_luhn/ 378282246310005 |execution_time|65.272µs |65.470µs ± 0.164µs |65.472µs ± 0.095µs |65.548µs |65.709µs |65.739µs |66.888µs |2.16% |3.297 |25.907 |0.25%|0.012µs |1 |200 | |credit_card/is_card_number_no_luhn/ 378282246310005 |throughput |14950377.937op/s |15274158.316op/s ± 37987.566op/s |15273709.366op/s ± 22166.521op/s |15299585.386op/s |15318678.524op/s |15320011.459op/s |15320610.818op/s |0.31% |-3.172 |24.500 |0.25%|2686.127op/s |1 |200 | |credit_card/is_card_number_no_luhn/37828224631 |execution_time|4.599µs |4.611µs ± 0.008µs |4.612µs ± 0.007µs |4.618µs |4.625µs |4.631µs |4.632µs |0.45% |0.277 |-0.798 |0.18%|0.001µs |1 |200 | |credit_card/is_card_number_no_luhn/37828224631 |throughput |215886090.974op/s|216853129.685op/s ± 393821.346op/s|216847783.055op/s ± 330482.428op/s|217240856.336op/s|217397900.981op/s|217423541.538op/s|217428765.014op/s|0.27% |-0.270 |-0.807 |0.18%|27847.374op/s|1 |200 | |credit_card/is_card_number_no_luhn/378282246310005 |execution_time|62.067µs |62.537µs ± 0.119µs |62.510µs ± 0.046µs |62.608µs |62.739µs |62.825µs |62.895µs |0.62% |-0.173 |1.756 |0.19%|0.008µs |1 |200 | |credit_card/is_card_number_no_luhn/378282246310005 |throughput |15899500.845op/s |15990672.323op/s ± 30366.907op/s |15997477.198op/s ± 11856.247op/s |16005538.717op/s |16039082.465op/s |16083465.878op/s |16111606.711op/s |0.71% |0.194 |1.793 |0.19%|2147.265op/s |1 |200 | |credit_card/is_card_number_no_luhn/37828224631000521389798|execution_time|58.861µs |58.890µs ± 0.023µs |58.889µs ± 0.016µs |58.902µs |58.933µs |58.950µs |58.996µs |0.18% |1.140 |1.661 |0.04%|0.002µs |1 |200 | |credit_card/is_card_number_no_luhn/37828224631000521389798|throughput |16950262.844op/s |16980761.200op/s ± 6580.768op/s |16981178.910op/s ± 4701.576op/s |16986189.379op/s |16987882.034op/s |16989084.140op/s |16989244.110op/s |0.05% |-1.138 |1.648 |0.04%|465.331op/s |1 |200 | |credit_card/is_card_number_no_luhn/x371413321323331 |execution_time|6.819µs |6.835µs ± 0.026µs |6.824µs ± 0.003µs |6.836µs |6.900µs |6.906µs |6.988µs |2.41% |2.497 |7.349 |0.38%|0.002µs |1 |200 | |credit_card/is_card_number_no_luhn/x371413321323331 |throughput |143104381.668op/s|146305951.652op/s ± 548040.369op/s|146548482.334op/s ± 70427.261op/s |146609923.977op/s|146639909.826op/s|146654171.210op/s|146657857.157op/s|0.07% |-2.463 |7.058 |0.37%|38752.306op/s|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |----------------------------------------------------------|--------------|------------------------------------------------------------|-------------------|------------------------|---------------| |credit_card/is_card_number/ |execution_time|[4.612µs; 4.614µs] or [-0.025%; +0.025%] |None |None |None | |credit_card/is_card_number/ |throughput |[216739024.301op/s; 216847627.155op/s] or [-0.025%; +0.025%]|None |None |None | |credit_card/is_card_number/ 3782-8224-6310-005 |execution_time|[93.457µs; 93.559µs] or [-0.054%; +0.054%] |None |None |None | |credit_card/is_card_number/ 3782-8224-6310-005 |throughput |[10688783.963op/s; 10700077.708op/s] or [-0.053%; +0.053%] |None |None |None | |credit_card/is_card_number/ 378282246310005 |execution_time|[85.345µs; 85.446µs] or [-0.059%; +0.059%] |None |None |None | |credit_card/is_card_number/ 378282246310005 |throughput |[11703700.248op/s; 11717161.511op/s] or [-0.057%; +0.057%] |None |None |None | |credit_card/is_card_number/37828224631 |execution_time|[4.611µs; 4.613µs] or [-0.026%; +0.026%] |None |None |None | |credit_card/is_card_number/37828224631 |throughput |[216766295.873op/s; 216880792.493op/s] or [-0.026%; +0.026%]|None |None |None | |credit_card/is_card_number/378282246310005 |execution_time|[81.945µs; 81.991µs] or [-0.029%; +0.029%] |None |None |None | |credit_card/is_card_number/378282246310005 |throughput |[12196457.468op/s; 12203408.382op/s] or [-0.028%; +0.028%] |None |None |None | |credit_card/is_card_number/37828224631000521389798 |execution_time|[58.887µs; 58.893µs] or [-0.005%; +0.005%] |None |None |None | |credit_card/is_card_number/37828224631000521389798 |throughput |[16979868.230op/s; 16981658.000op/s] or [-0.005%; +0.005%] |None |None |None | |credit_card/is_card_number/x371413321323331 |execution_time|[6.830µs; 6.837µs] or [-0.052%; +0.052%] |None |None |None | |credit_card/is_card_number/x371413321323331 |throughput |[146257078.847op/s; 146407106.275op/s] or [-0.051%; +0.051%]|None |None |None | |credit_card/is_card_number_no_luhn/ |execution_time|[4.611µs; 4.613µs] or [-0.024%; +0.024%] |None |None |None | |credit_card/is_card_number_no_luhn/ |throughput |[216764120.477op/s; 216867555.068op/s] or [-0.024%; +0.024%]|None |None |None | |credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 |execution_time|[73.678µs; 73.713µs] or [-0.024%; +0.024%] |None |None |None | |credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 |throughput |[13566212.518op/s; 13572610.971op/s] or [-0.024%; +0.024%] |None |None |None | |credit_card/is_card_number_no_luhn/ 378282246310005 |execution_time|[65.448µs; 65.493µs] or [-0.035%; +0.035%] |None |None |None | |credit_card/is_card_number_no_luhn/ 378282246310005 |throughput |[15268893.604op/s; 15279423.027op/s] or [-0.034%; +0.034%] |None |None |None | |credit_card/is_card_number_no_luhn/37828224631 |execution_time|[4.610µs; 4.613µs] or [-0.025%; +0.025%] |None |None |None | |credit_card/is_card_number_no_luhn/37828224631 |throughput |[216798549.834op/s; 216907709.536op/s] or [-0.025%; +0.025%]|None |None |None | |credit_card/is_card_number_no_luhn/378282246310005 |execution_time|[62.520µs; 62.553µs] or [-0.026%; +0.026%] |None |None |None | |credit_card/is_card_number_no_luhn/378282246310005 |throughput |[15986463.762op/s; 15994880.885op/s] or [-0.026%; +0.026%] |None |None |None | |credit_card/is_card_number_no_luhn/37828224631000521389798|execution_time|[58.887µs; 58.893µs] or [-0.005%; +0.005%] |None |None |None | |credit_card/is_card_number_no_luhn/37828224631000521389798|throughput |[16979849.168op/s; 16981673.231op/s] or [-0.005%; +0.005%] |None |None |None | |credit_card/is_card_number_no_luhn/x371413321323331 |execution_time|[6.832µs; 6.839µs] or [-0.052%; +0.052%] |None |None |None | |credit_card/is_card_number_no_luhn/x371413321323331 |throughput |[146229998.528op/s; 146381904.776op/s] or [-0.052%; +0.052%]|None |None |None | ### Group 3 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |-----------------------|--------------|-------|-----------------|-----------------|-------|-------|-------|-------|--------------------|--------|--------|-----|-------|----|-----------| |tags/replace_trace_tags|execution_time|2.656µs|2.733µs ± 0.025µs|2.733µs ± 0.005µs|2.739µs|2.790µs|2.797µs|2.798µs|2.40% |-0.411 |3.159 |0.93%|0.002µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |-----------------------|--------------|----------------------------------------|-------------------|------------------------|---------------| |tags/replace_trace_tags|execution_time|[2.730µs; 2.737µs] or [-0.129%; +0.129%]|None |None |None | ### Group 4 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |-------------------------------------------------------------------------------------------------------|--------------|----------------|--------------------------------|--------------------------------|----------------|----------------|----------------|----------------|--------------------|--------|--------|-----|------------|----|-----------| |normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...|execution_time|506.329µs |507.617µs ± 0.555µs |507.599µs ± 0.380µs |507.994µs |508.533µs |508.891µs |509.037µs |0.28% |0.021 |-0.420 |0.11%|0.039µs |1 |200 | |normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...|throughput |1964494.421op/s |1969991.281op/s ± 2154.950op/s |1970057.216op/s ± 1473.496op/s |1971294.247op/s |1973503.729op/s |1974293.356op/s |1974999.756op/s |0.25% |-0.016 |-0.422 |0.11%|152.378op/s |1 |200 | |normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて |execution_time|470.235µs |470.841µs ± 0.267µs |470.822µs ± 0.158µs |470.993µs |471.298µs |471.546µs |471.766µs |0.20% |0.460 |0.444 |0.06%|0.019µs |1 |200 | |normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて |throughput |2119693.042op/s |2123858.425op/s ± 1203.523op/s |2123946.474op/s ± 713.832op/s |2124654.901op/s |2125606.460op/s |2126392.240op/s |2126598.419op/s |0.12% |-0.457 |0.438 |0.06%|85.102op/s |1 |200 | |normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters |execution_time|179.886µs |180.321µs ± 0.174µs |180.304µs ± 0.120µs |180.440µs |180.624µs |180.719µs |180.796µs |0.27% |0.068 |-0.365 |0.10%|0.012µs |1 |200 | |normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters |throughput |5531101.044op/s |5545678.626op/s ± 5343.198op/s |5546191.661op/s ± 3703.943op/s |5549234.466op/s |5554642.238op/s |5556376.831op/s |5559078.565op/s |0.23% |-0.063 |-0.366 |0.10%|377.821op/s |1 |200 | |normalization/normalize_service/normalize_service/[empty string] |execution_time|44.807µs |44.996µs ± 0.065µs |44.989µs ± 0.045µs |45.049µs |45.101µs |45.115µs |45.148µs |0.35% |-0.101 |-0.352 |0.14%|0.005µs |1 |200 | |normalization/normalize_service/normalize_service/[empty string] |throughput |22149521.523op/s|22224241.241op/s ± 32008.737op/s|22227680.179op/s ± 22285.954op/s|22245974.051op/s|22275751.265op/s|22302952.160op/s|22317920.198op/s|0.41% |0.108 |-0.345 |0.14%|2263.359op/s|1 |200 | |normalization/normalize_service/normalize_service/test_ASCII |execution_time|48.962µs |49.214µs ± 0.119µs |49.190µs ± 0.051µs |49.271µs |49.349µs |49.400µs |50.379µs |2.42% |4.722 |44.263 |0.24%|0.008µs |1 |200 | |normalization/normalize_service/normalize_service/test_ASCII |throughput |19849452.412op/s|20319595.551op/s ± 48591.978op/s|20329369.267op/s ± 20878.509op/s|20343949.165op/s|20380932.354op/s|20401729.845op/s|20423987.565op/s|0.47% |-4.552 |42.116 |0.24%|3435.972op/s|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |-------------------------------------------------------------------------------------------------------|--------------|----------------------------------------------------------|-------------------|------------------------|---------------| |normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...|execution_time|[507.540µs; 507.694µs] or [-0.015%; +0.015%] |None |None |None | |normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000...|throughput |[1969692.626op/s; 1970289.937op/s] or [-0.015%; +0.015%] |None |None |None | |normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて |execution_time|[470.804µs; 470.878µs] or [-0.008%; +0.008%] |None |None |None | |normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて |throughput |[2123691.629op/s; 2124025.222op/s] or [-0.008%; +0.008%] |None |None |None | |normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters |execution_time|[180.297µs; 180.345µs] or [-0.013%; +0.013%] |None |None |None | |normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters |throughput |[5544938.110op/s; 5546419.142op/s] or [-0.013%; +0.013%] |None |None |None | |normalization/normalize_service/normalize_service/[empty string] |execution_time|[44.987µs; 45.005µs] or [-0.020%; +0.020%] |None |None |None | |normalization/normalize_service/normalize_service/[empty string] |throughput |[22219805.137op/s; 22228677.344op/s] or [-0.020%; +0.020%]|None |None |None | |normalization/normalize_service/normalize_service/test_ASCII |execution_time|[49.197µs; 49.230µs] or [-0.034%; +0.034%] |None |None |None | |normalization/normalize_service/normalize_service/test_ASCII |throughput |[20312861.170op/s; 20326329.932op/s] or [-0.033%; +0.033%]|None |None |None | ### Group 5 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |----------------------------|--------------|--------|------------------|------------------|--------|--------|--------|--------|--------------------|--------|--------|-----|-------|----|-----------| |redis/obfuscate_redis_string|execution_time|36.961µs|37.536µs ± 0.974µs|37.084µs ± 0.048µs|37.210µs|39.626µs|39.678µs|40.391µs|8.92% |1.700 |0.966 |2.59%|0.069µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |----------------------------|--------------|------------------------------------------|-------------------|------------------------|---------------| |redis/obfuscate_redis_string|execution_time|[37.401µs; 37.671µs] or [-0.360%; +0.360%]|None |None |None | ### Group 6 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |----------------------------------------|--------------|---------|--------------------|-------------------|---------|---------|---------|---------|--------------------|--------|--------|-----|-------|----|-----------| |normalization/normalize_trace/test_trace|execution_time|293.960ns|305.108ns ± 13.764ns|299.093ns ± 3.320ns|305.715ns|338.107ns|340.982ns|341.786ns|14.27% |1.615 |1.135 |4.50%|0.973ns|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |----------------------------------------|--------------|--------------------------------------------|-------------------|------------------------|---------------| |normalization/normalize_trace/test_trace|execution_time|[303.200ns; 307.015ns] or [-0.625%; +0.625%]|None |None |None | ### Group 7 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |--------------------------------------|--------------|-------|-----------------|-----------------|-------|-------|-------|-------|--------------------|--------|--------|-----|-------|----|-----------| |concentrator/add_spans_to_concentrator|execution_time|9.015ms|9.056ms ± 0.030ms|9.055ms ± 0.010ms|9.063ms|9.076ms|9.127ms|9.351ms|3.27% |6.661 |58.961 |0.33%|0.002ms|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |--------------------------------------|--------------|----------------------------------------|-------------------|------------------------|---------------| |concentrator/add_spans_to_concentrator|execution_time|[9.052ms; 9.060ms] or [-0.045%; +0.045%]|None |None |None | ### Group 8 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |-------------------------------------------------------------------------------------------------------|--------------|----------------|---------------------------------|---------------------------------|----------------|----------------|----------------|----------------|--------------------|--------|--------|-----|-------------|----|-----------| |normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...|execution_time|271.954µs |273.821µs ± 1.039µs |273.698µs ± 0.757µs |274.491µs |275.673µs |276.054µs |277.168µs |1.27% |0.485 |-0.356 |0.38%|0.073µs |1 |200 | |normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...|throughput |3607919.238op/s |3652070.319op/s ± 13833.931op/s |3653655.613op/s ± 10101.367op/s |3662831.367op/s |3670673.291op/s |3676722.795op/s |3677099.210op/s |0.64% |-0.469 |-0.383 |0.38%|978.207op/s |1 |200 | |normalization/normalize_name/normalize_name/bad-name |execution_time|25.814µs |26.307µs ± 0.421µs |26.055µs ± 0.168µs |26.745µs |26.953µs |27.068µs |27.227µs |4.50% |0.434 |-1.532 |1.60%|0.030µs |1 |200 | |normalization/normalize_name/normalize_name/bad-name |throughput |36727743.417op/s|38022402.340op/s ± 604973.511op/s|38379867.566op/s ± 249644.919op/s|38544741.484op/s|38664550.533op/s|38699379.141op/s|38738542.714op/s|0.93% |-0.421 |-1.550 |1.59%|42778.087op/s|1 |200 | |normalization/normalize_name/normalize_name/good |execution_time|15.446µs |15.609µs ± 0.089µs |15.604µs ± 0.073µs |15.682µs |15.745µs |15.799µs |15.828µs |1.44% |0.132 |-0.941 |0.57%|0.006µs |1 |200 | |normalization/normalize_name/normalize_name/good |throughput |63178344.569op/s|64069522.759op/s ± 365369.611op/s|64087757.669op/s ± 298839.450op/s|64349434.738op/s|64633866.955op/s|64695637.209op/s|64742938.164op/s|1.02% |-0.115 |-0.955 |0.57%|25835.533op/s|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |-------------------------------------------------------------------------------------------------------|--------------|----------------------------------------------------------|-------------------|------------------------|---------------| |normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...|execution_time|[273.677µs; 273.965µs] or [-0.053%; +0.053%] |None |None |None | |normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo...|throughput |[3650153.069op/s; 3653987.569op/s] or [-0.052%; +0.052%] |None |None |None | |normalization/normalize_name/normalize_name/bad-name |execution_time|[26.249µs; 26.365µs] or [-0.222%; +0.222%] |None |None |None | |normalization/normalize_name/normalize_name/bad-name |throughput |[37938558.830op/s; 38106245.850op/s] or [-0.221%; +0.221%]|None |None |None | |normalization/normalize_name/normalize_name/good |execution_time|[15.596µs; 15.621µs] or [-0.079%; +0.079%] |None |None |None | |normalization/normalize_name/normalize_name/good |throughput |[64018886.045op/s; 64120159.474op/s] or [-0.079%; +0.079%]|None |None |None | ### Group 9 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |--------------------|--------------|-------|-----------------|-----------------|-------|-------|--------|--------|--------------------|--------|--------|------|-------|----|-----------| |write only interface|execution_time|1.357µs|3.294µs ± 1.415µs|3.145µs ± 0.023µs|3.166µs|3.197µs|14.070µs|15.100µs|380.11% |7.547 |57.275 |42.86%|0.100µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |--------------------|--------------|----------------------------------------|-------------------|------------------------|---------------| |write only interface|execution_time|[3.097µs; 3.490µs] or [-5.955%; +5.955%]|None |None |None | ### Group 10 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |----------------------------------------------|--------------|---------|-------------------|-------------------|---------|---------|---------|---------|--------------------|--------|--------|-----|-------|----|-----------| |benching string interning on wordpress profile|execution_time|140.534µs|141.600µs ± 0.954µs|141.522µs ± 0.182µs|141.697µs|142.224µs|142.600µs|154.125µs|8.91% |11.402 |147.164 |0.67%|0.067µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |----------------------------------------------|--------------|--------------------------------------------|-------------------|------------------------|---------------| |benching string interning on wordpress profile|execution_time|[141.467µs; 141.732µs] or [-0.093%; +0.093%]|None |None |None | ### Group 11 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |-----------------|--------------|--------|-------------------|------------------|--------|--------|--------|---------|--------------------|--------|--------|------|-------|----|-----------| |two way interface|execution_time|19.335µs|26.909µs ± 13.684µs|20.866µs ± 0.713µs|35.194µs|46.758µs|93.313µs|131.532µs|530.35% |3.994 |22.643 |50.73%|0.968µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |-----------------|--------------|------------------------------------------|-------------------|------------------------|---------------| |two way interface|execution_time|[25.012µs; 28.805µs] or [-7.048%; +7.048%]|None |None |None | ### Group 12 |cpu_model |git_commit_sha|git_commit_date|git_branch | |----------------------------------------------|--------------|---------------|--------------------| |Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz|3cd888a3 |1731617129 |dsn/reduce-fuzz-time| |scenario |metric |min |mean ± sd |median ± mad |p75 |p95 |p99 |max |peak_to_median_ratio|skewness|kurtosis|cv |sem |runs|sample_size| |------------------------|--------------|--------|------------------|------------------|--------|--------|--------|--------|--------------------|--------|--------|-----|-------|----|-----------| |sql/obfuscate_sql_string|execution_time|70.215µs|70.381µs ± 0.154µs|70.359µs ± 0.030µs|70.386µs|70.541µs|70.703µs|72.160µs|2.56% |8.361 |89.152 |0.22%|0.011µs|1 |200 | |scenario |metric |95% CI mean |Shapiro-Wilk pvalue|Ljung-Box pvalue (lag=1)|Dip test pvalue| |------------------------|--------------|------------------------------------------|-------------------|------------------------|---------------| |sql/obfuscate_sql_string|execution_time|[70.360µs; 70.402µs] or [-0.030%; +0.030%]|None |None |None |Baseline
Omitted due to size.
codecov-commenter
What does this PR do?
Reduces the time per fuzz test from 1min -> 30s for PRs.
Motivation
The "profiling" fuzz tests take 18 min. This is over our desired 10min CI budget.
Additional Notes
This lowers coverage. We should add a nightly job that runs the latest main for longer to get coverage back.
How to test the change?
Describe here in detail how the change can be validated.