Open remittor opened 1 year ago
Parsed stats for frameworks:
Format: total_test_time
(build_time
+ verify_time
) framework_name
$ python3 tfb_res.py --fw -f results.2023-04-06-08-48-50-340.zip
Number of frameworks: 295
Number of tests: 767
Total time: 6 days, 15:26:25 (included 12.78 hours of sleep)
Total time: 146.66 + 12.78 = 159.44 hours
Total build time: 27.65 hours
Total verify time: 114.96 hours
Verify time avg: (114.96 * 60) / 767 = 8.99 min
Number of full tests: 312
Full tests verify time avg: (68.03 * 60) / 312 = 13.08 min
3:32:39 (0:16:58 + 3:12:51) http4k
2:52:36 (0:09:14 + 2:40:22) officefloor
2:25:48 (0:39:08 + 1:42:56) ubiquity
2:17:17 (0:54:02 + 1:17:45) cutelyst
2:16:51 (0:23:52 + 1:49:07) php
2:12:59 (0:08:39 + 2:01:38) rack-sequel
2:04:03 (0:17:55 + 1:43:27) falcon
2:02:45 (0:14:35 + 1:46:33) hexagon
1:57:36 (0:28:35 + 1:24:59) micronaut ERROR: "Problem starting micronaut-graalvm"
1:51:30 (0:11:01 + 1:35:05) play2
1:51:19 (0:08:19 + 1:40:15) sinatra-sequel
1:48:03 (0:08:58 + 1:33:13) pippo
1:47:13 (0:07:50 + 1:36:32) fastapi
1:43:48 (0:17:06 + 1:24:16) imi
1:42:50 (0:07:54 + 1:31:18) nestjs
1:41:11 (0:12:22 + 1:26:59) ktor ERROR: "Problem starting ktor-exposed-dao"
1:40:52 (0:31:50 + 1:05:01) spring
1:38:35 (0:35:36 + 1:01:02) treefrog
1:38:24 (0:58:54 + 0:37:34) spliffy
1:31:33 (0:45:54 + 0:42:07) axum
1:29:46 (0:02:07 + 1:25:01) chi
1:28:21 (0:21:26 + 1:05:01) laravel
1:26:45 (0:06:51 + 1:17:58) roda-sequel
1:26:34 (0:46:46 + 0:38:57) warp
1:20:54 (0:34:48 + 0:43:50) actix
1:19:36 (0:07:30 + 1:10:09) sinatra
1:16:44 (0:23:15 + 0:52:01) lumen
1:14:00 (0:02:25 + 1:09:07) ffead-cpp
1:09:10 (0:01:16 + 1:04:12) nodejs
1:06:18 (0:28:01 + 0:37:30) appmpower
1:05:01 (0:03:00 + 1:00:56) jooby
1:04:46 (0:11:37 + 0:52:22) viz
1:03:35 (0:14:23 + 0:47:55) mixphp
0:59:41 (0:28:26 + 0:29:53) ntex ERROR: "Problem starting ntex-plt-astd"
0:59:11 (0:01:35 + 0:55:24) hapi
0:56:06 (0:07:04 + 0:45:33) vertx-web ERROR: "Framework is not accepting requests from client machine"
0:54:50 (0:09:39 + 0:43:55) kumbiaphp
0:54:04 (0:31:45 + 0:21:56) drogon
0:54:00 (0:05:10 + 0:45:30) lithium
0:52:53 (0:44:00 + 0:08:32) servant
0:52:28 (0:02:45 + 0:48:44) microdot
0:51:16 (0:13:51 + 0:36:06) roa
0:49:57 (0:03:48 + 0:45:28) redkale
0:47:44 (0:15:51 + 0:30:55) swoole
0:47:43 (0:07:07 + 0:39:48) xitca-web
0:46:54 (0:22:56 + 0:21:59) vapor
0:46:31 (0:14:36 + 0:30:55) php-ngx
0:46:13 (0:14:20 + 0:30:57) openswoole
0:44:53 (0:05:02 + 0:38:57) quarkus
0:44:28 (0:08:34 + 0:34:55) slim
Rust framework always much time on build docker images . Is taht possible to use docker muilt-build stage or shared a comon build image or other way to share the compile results?
Let's take a look at this:
0:33:28 (0:02:48 + 0:30:27) viz-diesel
Verify time = 30.5 min (avg time = 13 min)
Let's look at the details for test fortunes
:
https://tfb-status.techempower.com/unzip/results.2023-04-06-08-48-50-340.zip/results/20230330235259/viz-diesel/fortune
Concurrency: 16 for fortune
wrk -H 'Host: 10.0.0.1' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 16 --timeout 8 -t 16 "http://10.0.0.1:8080/fortunes"
---------------------------------------------------------
unable to connect to 10.0.0.1:8080 Connection timed out
STARTTIME 1680761646
ENDTIME 1680761776
All fortunes
tests failed and took a lot of time: 776 - 646 = 130 sec (2.17 min)
2.17 min * 8 tests = 17.33 min
Checking the result by the time the file was created stat.txt
and raw.txt
:
1680751642 - 1680750576 = 1066 sec = 17.76 min
Ideally, each test should take 15 seconds.
So the ideal total time for fortune
test should be: 15 * 8 = 120 seconds (2 min).
But viz-diesel
fortunes
test somehow managed to pass the test:
--------------------------------------------------------------------------------
VERIFYING FORTUNE
--------------------------------------------------------------------------------
PASS for http://10.0.0.1:8080/fortunes
Executed queries: 512/512
PASS for http://10.0.0.1:8080/fortunes
Rows read: 6077/6144
Test pre-verification time: 30.5 min - 17.76 min = 12.74 min
Verify test in CI: https://github.com/TechEmpower/FrameworkBenchmarks/actions/runs/4554118663/jobs/8031564280#step:9:876
UPDATE Full stat for this test:
$ python3 tfb_res.py -f results.2023-04-06-08-48-50-340.zip -t viz-diesel
Number of frameworks: 295
Number of tests: 767
Total time: 6 days, 15:26:25 (included 12.78 hours of sleep)
Total time: 146.66 + 12.78 = 159.44 hours
Total build time: 27.65 hours
Total verify time: 114.96 hours
Verify time avg: (114.96 * 60) / 767 = 8.99 min
Number of full tests: 316
Full tests verify time avg: (68.86 * 60) / 316 = 13.07 min
0:33:28 (0:02:48 + 0:30:27) viz-diesel
verify test
db : 0:00:04 0:02:12
query : 0:00:16 0:01:54
update : 0:00:22 0:07:42
fortune : 0:00:04 0:17:46
TOTAL : 0:00:46 0:29:34
Total verify and test time: 0:30:20
Let's take a look at this:
0:24:15 (0:01:18 + 0:22:42) rack-sequel-postgres-passenger-mri
Verify time = 22.7 min (avg time = 13 min)
Let's look at the details for test fortunes: https://tfb-status.techempower.com/unzip/results.2023-04-06-08-48-50-340.zip/results/20230330235259/rack-sequel-postgres-passenger-mri
Full stat for this test:
$ python3 tfb_res.py -f results.2023-04-06-08-48-50-340.zip -t rack-sequel-postgres-passenger-mri
Number of frameworks: 295
Number of tests: 767
Total time: 6 days, 15:26:25 (included 12.78 hours of sleep)
Total time: 146.66 + 12.78 = 159.44 hours
Total build time: 27.65 hours
Total verify time: 114.96 hours
Verify time avg: (114.96 * 60) / 767 = 8.99 min
Number of full tests: 316
Full tests verify time avg: (68.86 * 60) / 316 = 13.07 min
0:24:15 (0:01:18 + 0:22:42) rack-sequel-postgres-passenger-mri
verify test
db : 0:00:06 0:04:04
json : 0:00:04 0:05:22
query : 0:00:16 0:02:58
update : 0:00:16 0:03:00
fortune : 0:00:04 0:03:54
plaintext : 0:00:04 0:02:24
TOTAL : 0:00:50 0:21:42
Total verify and test time: 0:22:32
Look to /json/raw.txt
: https://tfb-status.techempower.com/unzip/results.2023-04-06-08-48-50-340.zip/results/20230330235259/rack-sequel-postgres-passenger-mri/json/raw.txt
Concurrency: 256 for json
wrk -H 'Host: 10.0.0.1' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 256 --timeout 8 -t 28 "http://10.0.0.1:8080/json"
---------------------------------------------------------
Running 15s test @ http://10.0.0.1:8080/json
28 threads and 256 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 6.12ms 1.02ms 11.65ms 79.87%
Req/Sec 1.12k 159.09 1.26k 89.29%
Latency Distribution
50% 6.13ms
75% 6.67ms
90% 7.26ms
99% 8.76ms
3110 requests in 15.09s, 735.41KB read
Requests/sec: 206.16
Transfer/sec: 48.75KB
STARTTIME 1680629011
ENDTIME 1680629091
1680629091 - 1680629011 = 80 sec
@remittor where did you get tfb_res.py
? I can't find it in the repository.
I'm looking the way to output total stats for results of tests started in my local environment. Ideally if it will be possible to view a comparison between two runs. Is there such tool?
@svetlyak40wt , look first post https://gist.github.com/remittor/38ef89e47c18f6ec2a2095bc7390cdf3
TFB results of 2023-04-06: https://www.techempower.com/benchmarks/#section=test&runid=ab65d6b5-8efc-4a65-9c5c-c31667418528
Full stat of tests: https://tfb-status.techempower.com/raw/results.2023-04-06-08-48-50-340.zip (56MB)
Parsed stats: Format:
total_test_time
(build_time
+verify_time
)test_name
Parser source code: https://gist.github.com/remittor/38ef89e47c18f6ec2a2095bc7390cdf3