Memory layout differences can render benchmark results meaningless

hwayne / awesome-cold-showers

For when people get too hyped up about things

Other

7.26k stars 82 forks source link

Memory layout differences can render benchmark results meaningless #15

Open kqr opened 4 years ago

kqr commented 4 years ago

In the first half of Performance Matters (video) the speaker talks about how big effects memory layout can have on the results of performance benchmarks.

What's worse is that this is a variable that can fluctuate rather wildly for no obvious reason. I don't remember how well the research was made, so that would remain to be vetted.

hwayne commented 4 years ago

We happen to be acquainted, I'll see if he wants to do a writeup

emeryberger commented 4 years ago

I'd say pretty well vetted. Two ASPLOS papers, one by my student @ccurtsinger and myself [ASPLOS'13], reproducing results by from Todd Mytkowicz et al [ASPLOS'09].

https://users.cs.northwestern.edu/~robby/courses/322-2013-spring/mytkowicz-wrong-data.pdf https://people.cs.umass.edu/~emery/pubs/stabilizer-asplos13.pdf

kqr commented 4 years ago

Thank you for the references and the great presentation, @emeryberger !

hwayne commented 1 year ago

I'll try to write it up later this month m'self