mkeeter
commented
3 months ago A few other observations:
BRM42220058 has less RAM than the other machines (917478 MiB versus 1036256 MiB). It's hard to imagine how a 12% difference would cause such a dramatic performance change, though.
The slow sled is also doing slightly more writes per call to nvme_bd_write. Looking at avg(args[1]->xblks) in ::nvme_bd_write:entry, a fast sled is touching 246 blocks, versus 304 for the slow sled.
Many things in the slow sled are slower; for example, running nvmeadm list is about 600 ms slower. Individual functions to build and submit jobs are slower, e.g. nvme_bd_write → nvme_create_nvm_cmd → nvme_fill_prp are all roughly 2x slower on the bad sled (measuring from :entry to :return).
jclulow
faithanalog
Using iodriver and construct-additional-pylons, we've been doing performance benchmarking on the London mini-rack (2x switches, 4x Gimlets). Most recently, we did a group debugging session with 64x VMs, all doing 4K random writes to their own 64 GiB disk (structured as 4 KiB block size, 16384 blocks/extent, 1024 extents, encrypted).
During the meeting, we observed high CPU load and mediocre IOPS (120-160). Here are my notes from right after the meeting:
For more see the recording, transcript, and chat.
The "flamegraph" discussed in these observations is shown below. It was captured using
flamegraph -c'profile-997 { @[stack(100)] = count(); } -p1(the PID is a dummy value, because we're profiling the kernel here).Further investigation outside the scope of the meeting revealed bimodal performance:
(the X axis isn't particularly meaningful here)
We found that one of the sleds had much less IO throughput than the other three – specifically, every Upstairs with a Downstairs on BRM42220058 is slow.
IOPS as a function of Downstairs locations:
This slow sled happens to be the one from which we took the flamegraph above.
Here's an example of a flamegraph from a non-slow sled:
We can see the slower performance just by running
iostat -x 1 2(looking at the second sample printed):There's not a dramatically mismatched distribution of VMs per sled or regions per disk:
Using DTrace, I measured NVMe write performance relatively close to the metal. I found that the slow sled has noticeably longer delay between
nvme_bd_writeandbd_xfer_done:Weirdly enough, this slowdown isn't on a single drive; it's noticeable on every single NVMe drive on BRM42220058. Splitting them out by NVMe namespace (on just the slow sled), here's what we see:
Taking the averages instead, we can compare time between
nvme_bd_writeandbd_xfer_doneacross all four sleds. The slow sled is a clear outlier.