search

TechEmpower / FrameworkBenchmarks

Source for the TechEmpower Framework Benchmarks project
https://www.techempower.com/benchmarks/
Other
7.54k stars 1.93k forks source link

reported results are not feasible #9168

Open robaho opened 1 month ago

robaho commented 1 month ago

The vast majority of the plaintext results are not possible on 10 Gbit ethernet.

The maximum transfer speed of 10gbit ethernet is 1,250 megabytes per second.

The minimum valid http response for plaintext is 142 bytes, the headers are 129 bytes, and the payload 13 bytes.

HTTP/1.1 200 OK
Server: httpserver-robaho
Date: Tue, 16 Jul 2024 14:35:23 GMT
Content-type: text/plain
Content-length: 13

You could reduce it a bit will a smaller server name.

so, the top result of 28,733,734 by mrhttp is not possible, as this would equate to:

28733734 142 / (10241024) = 3891 megabytes per second

The approximate theoretical maximum requests per second between a single client and single server machine on 10gbit ethernet is 9230422 requests per second - which would mean everything above 70th place is impossible. This also assumes that the packet rate is not exceeded (i.e. that every ethernet frame is filled to capacity).

The only way this would be possible is using multiple coordinated client machines with dedicated ethernet connections to multiple server machines - but the latest environment documentation does not describe this as being the case.

Can you provide more details on how these numbers are achieved? Thanks.

robaho commented 1 month ago

I think I see the issue. I don't think the website is working properly. I think it is showing cloud results for the physical as they are the same.

image image
robaho commented 1 month ago

Strangely though, the official benchmarks do not show any results for Cloud for plaintext or json serialization.

NateBrady23 commented 1 month ago

There is currently no cloud environment.

The current environment is 40gbit.

Have to update the docs.

robaho commented 1 month ago

There is currently no cloud environment.

The current environment is 40gbit.

Have to update the docs.

When you click cloud, the results should probably be empty then to not confuse people.

remittor commented 1 month ago

Can you provide more details on how these numbers are achieved? Thanks.

There are many explanations here: https://github.com/TechEmpower/FrameworkBenchmarks/issues/7402

Here I repeated the results on 99% identical hardware for the libreactor framework: https://github.com/TechEmpower/FrameworkBenchmarks/issues/7402#issuecomment-1482918369

There I also showed the throughput when using a 25Gbit network: 21387110 req/sec

robaho commented 1 month ago

Can you provide more details on how these numbers are achieved? Thanks.

There are many explanations here: #7402

Here I repeated the results on 99% identical hardware for the libreactor framework: #7402 (comment)

There I also showed the throughput when using a 25Gbit network: 21387110 req/sec

Understood. The docs stated it was 10gbit Ethernet which is what made them infeasible.