Closed mkeeter closed 2 months ago
A few other observations:
BRM42220058 has less RAM than the other machines (917478 MiB versus 1036256 MiB). It's hard to imagine how a 12% difference would cause such a dramatic performance change, though.
The slow sled is also doing slightly more writes per call to nvme_bd_write
. Looking at avg(args[1]->xblks)
in ::nvme_bd_write:entry
, a fast sled is touching 246 blocks, versus 304 for the slow sled.
Many things in the slow sled are slower; for example, running nvmeadm list
is about 600 ms slower. Individual functions to build and submit jobs are slower, e.g. nvme_bd_write
→ nvme_create_nvm_cmd
→ nvme_fill_prp
are all roughly 2x slower on the bad sled (measuring from :entry
to :return
).
I have replaced BRM42220058 with 0XV1:9130000019:006:BRM42220041 which was at least just able to go through a full programming cycle on zoot.
This was solved by https://github.com/oxidecomputer/meta/issues/418
Using iodriver and construct-additional-pylons, we've been doing performance benchmarking on the London mini-rack (2x switches, 4x Gimlets). Most recently, we did a group debugging session with 64x VMs, all doing 4K random writes to their own 64 GiB disk (structured as 4 KiB block size, 16384 blocks/extent, 1024 extents, encrypted).
During the meeting, we observed high CPU load and mediocre IOPS (120-160). Here are my notes from right after the meeting:
For more see the recording, transcript, and chat.
The "flamegraph" discussed in these observations is shown below. It was captured using
flamegraph -c'profile-997 { @[stack(100)] = count(); } -p1
(the PID is a dummy value, because we're profiling the kernel here).Further investigation outside the scope of the meeting revealed bimodal performance:
(the X axis isn't particularly meaningful here)
We found that one of the sleds had much less IO throughput than the other three – specifically, every Upstairs with a Downstairs on BRM42220058 is slow.
IOPS as a function of Downstairs locations:
This slow sled happens to be the one from which we took the flamegraph above.
Here's an example of a flamegraph from a non-slow sled:
We can see the slower performance just by running
iostat -x 1 2
(looking at the second sample printed):There's not a dramatically mismatched distribution of VMs per sled or regions per disk:
Using DTrace, I measured NVMe write performance relatively close to the metal. I found that the slow sled has noticeably longer delay between
nvme_bd_write
andbd_xfer_done
:Weirdly enough, this slowdown isn't on a single drive; it's noticeable on every single NVMe drive on BRM42220058. Splitting them out by NVMe namespace (on just the slow sled), here's what we see:
Taking the averages instead, we can compare time between
nvme_bd_write
andbd_xfer_done
across all four sleds. The slow sled is a clear outlier.