search

TechEmpower / FrameworkBenchmarks

Source for the TechEmpower Framework Benchmarks project
https://www.techempower.com/benchmarks/
Other
7.63k stars 1.94k forks source link

Test for efficiency: Requests Per Second / Watt #4117

Open naturallymitchell opened 6 years ago

naturallymitchell commented 6 years ago

This test would help those looking to deploy hardware and software on a budget. Searching Google for "requests per second per watt" yields many results like this Register processor review. AFAIR, an Intel chief said that the most important measure of a processor wasn't performance, but performance per watt. Perhaps a software-defined power meter like powerapi could help.

matbrgz commented 6 years ago

I think this question is very valid, here in my university I already hear discussions about it. But nothing very profitable. To implement the PowerAPI next to the Benchmarks of the Frameworks it is necessary a discussion about it, so I understood PowerAPI would run alongside the framework, being necessary an implementation for each tested framework which today are more than 300 (if I am not mistaken), it runs in Scala then it is also necessary to reshape all docker file. I think it would be very valuable to take this discussion forward, I'm sure TechEmpower community?

naturallymitchell commented 6 years ago

It looks like s-tui CPU monitoring utility supports collecting power consumption data from any Intel CPU after Sandy Bridge, eg the Skylake CPUs in TechEmpower's servers, and can export it to csv.

bhauer commented 6 years ago

It's fun to see this idea come up here. We've tossed similar ideas around internally but haven't put the time into putting the necessary math together.

We even (mostly jokingly) toyed with working the math all of the way through to CO2 emissions from electricity generation in a given data center market, then mapping that to anything we can find from the IPCC. It would be fun to know your application server choice means 0.00000001 C per year (or whatever it works out to).

bhauer commented 5 years ago

"Your Software’s Carbon Footprint" is a blog entry/article discussing the notion I raised in my previous comment.

GaelGirodon commented 2 years ago

What about trying to use Scaphandre during benchmarks to measure power consumption? This tool looks promising, modern and optimized (written in Rust). This could be a good addition to existing metrics (RPS, latency, memory, CPU, ...) and could make these benchmarks even more relevant as more and more people are looking for energy and cost efficiency more than raw performance.

franz1981 commented 2 years ago

Great idea! Carbon could be interesting and relevant too!