obilaniu
commented
3 years ago @goldsteinn I wrote this small project for the purposes of some ultra-low-latency/overhead benchmarking on my own computer's CPU, an Intel Haswell microarchitecture processor (w/ perfmon v3). The intention was to bypass system calls by making the counters readable from userspace, and to configure them from inside the program too (but outside the benchmarked code).
Because there is seemingly no way to ask the Linux kernel's performance-monitoring subsystem what mapping of "events" to "counters" it applied, and there are important restrictions (not all events can be counted on every counter), I also gave this kernel module the ability to configure the counters itself. This configuration only really works on a single core (thus the pfcPinThread() call), but I had considered that acceptable because the microbenchmarking I had been doing is single-threaded performance anyways. Furthermore, there was no easy way to migrate counter values with the process when it is rescheduled to a different core except by using the Linux kernel's performance-monitoring subsystem.
The fact that this code configures the counters itself had several important consequences. It needs detailed knowledge of, and read/write access to, several CPU MSRs (model-specific registers), in order to safely program the perfmon subsystem. By delving deep into the Intel SDM, I was able to write that code for my own (Intel Haswell) CPU. The MSR addresses I write to are hardcoded in libpfcmsr.h, and the table of events and their codes for Haswell is in libpfc.c.
Every change of the perfmon architecture version puts the MSR numbering and event codes in danger. As it turns out the v4 was still somewhat backwards compatible, so I allowed it. But v5 seems to have broken it (as in here).
I now realize that the approach I took (baking in the constants for Haswell as C source code) was never going to scale in the long term. It was enormously labour-intensive, but worthwhile when I was working only on my own CPU. Scaling it to more CPU microarchitectures means writing CPU-detection code and more and more logic for every CPU out there. I can't do it, nor test it, when I only have one CPU model available and limited time.
If I had to do this project again, I would probably use libpfm, as suggested by Travis Downs, or maybe JSON/YAML files with the counters and events in declarative form. Or maybe I would simply accept using perf, PAPI, and other high-overhead APIs, but with very high loop trip counts.
As a post-script, there is a further complication you might be hitting here. The trend within the Linux kernel is, more and more, to control and lock things down. This includes MSRs, the access to which in recent Linux kernel releases has been tightened. In general, performance monitoring components are also security risks because they're side-channels through which valuable information can leak (e.g. speculative data fetch/control flow, Meltdown, Spectre, ...).
So I am being strangled by newer Linux kernels, which would rather not give low-overhead, unprivileged, read/write access to performance counters and their configuration. I can't easily fight that. These days, I usually use perf. This project is in some sense a vestige, a futile effort in light of the clear will of Linux kernel developers to force the usage of their entry points. But as a non-trivial Linux kernel module + userspace program pair, it still has some educational value, for others as well as myself, so I leave it online. It's my most-starred open-source project, for some reason.
goldsteinn
Hi,
Tried to use the module and got the following dmsg:
Was able to get the module running just changing https://github.com/obilaniu/libpfc/blob/master/kmod/pfckmod.c#L930 to include arch version 5 and run demo and obv there where some issues. The output of pfcdemo seemed fine but dmesg told another story.
Heres the dmesg:
Wondering if 1) How plan to add support for arch 5. (figure you have all the info from dmesg if you choose to do this. From the comments looks like this was written for skylake and earlier and I'm on icelake-client so maybe thats it. if so probably not too much to add support. if your not particularly interested I might try out a PR) 2) How serious is this error message in dmesg towards not causing kernel panics and towards the quality of the bench marks
Thanks.