mfence vs lfence vs cpuid

We've been a bit lazy on how we're using RDTSC. The original piece of code (probably about 10 years ago) had this comment:

Intel actually recommends calling CPUID to serialize the execution flow
 and reduce variance in measurement due to out-of-order execution.
 We don't do that here yet.
 see §3.2.1 http://www.intel.com/content/www/us/en/embedded/training/ia-32-ia-64-benchmark-code-execution-paper.html

That link is gone, but the paper can be found in mirrors. It's a good resource and has the following advice. We should probably just follow it:

Resources:

https://github.com/oreparaz/dudect/pull/30
This is how @dgruss does it: https://github.com/IAIK/cache_template_attacks/blob/main/cacheutils.h#L24-L31

uint64_t rdtsc() {
  uint64_t a, d;
  asm volatile ("mfence");
  asm volatile ("rdtsc" : "=a" (a), "=d" (d));
  a = (d<<32) | a;
  asm volatile ("mfence");
  return a;
}

This is how libcpucycles does it: https://cpucycles.cr.yp.to/libcpucycles-20230115/cpucycles/amd64-tscasm.c.html

long long ticks(void)
{
  unsigned long long result;
  asm volatile(".byte 15;.byte 49;shlq $32,%%rdx;orq %%rdx,%%rax"
    : "=a"(result) :: "%rdx");
  return result;
}

And the motivated reader can go thru Agner Fog's tools and see: https://www.agner.org/optimize/

The test programs use the serializing instruction CPUID before and after reading the time stamp counter in order to prevent out-of-order execution to interfere with the measurements.

oreparaz / dudect

mfence vs lfence vs cpuid #32