Investigate SSD R/W performance

[geoffxy]

We need to get a good understanding of how NVMe SSDs behave under read/write workloads. For example, is it possible to achieve the SSD's advertised peak sequential read and write bandwidths simultaneously? The answer to this question will influence TreeLine's design and/or implementation.

We should use fio for our benchmarking. We need to test out the following drive configurations

• Writing to the block device directly (i.e., no file system)
• Different file systems (ext4, XFS, btrfs)

We should try the following experiments and see what read/write bandwidth we can achieve:

• 50/50 random read random write (writes are overwrites)
• N threads making random reads, N threads making random writes (i.e., no artificial limitation on a 50/50 R/W mix)

We may also want to consider the effect of different I/O paths:

• Synchronous I/O (pwrite() / pread())
• io_uring
• Linux libaio

[geoffxy]

Here are my results from earlier in the project. I looked at random R/W workloads running on top of ext4. Unfortunately I did not record the block size used, so these results should only guide our investigation.

Running a 50/50 R/W workload:

Run status group 3 (all jobs):
   READ: bw=685MiB/s (718MB/s), 685MiB/s-685MiB/s (718MB/s-718MB/s), io=1021MiB (1070MB), run=1491-1491msec
  WRITE: bw=689MiB/s (722MB/s), 689MiB/s-689MiB/s (722MB/s-722MB/s), io=1027MiB (1077MB), run=1491-1491msec

Note: Because it's 50/50, the achieved read/write bandwidths are expected to be equal. This is because the number of reads and writes must be equal, so the bandwidths will match the slower of read/write.

Running an independent read/write workload (reader just doing random reads, writer doing random writes):

Run status group 0 (all jobs):
   READ: bw=1085MiB/s (1138MB/s), 1085MiB/s-1085MiB/s (1138MB/s-1138MB/s), io=31.8GiB (34.1GB), run=30005-30005msec
  WRITE: bw=552MiB/s (579MB/s), 552MiB/s-552MiB/s (579MB/s-579MB/s), io=16.2GiB (17.4GB), run=30004-30004msec

[geoffxy]

Key Takeaways for an Intel DC P4510 NVMe SSD (1 TB)

Write-only

• It is possible to achieve the device's peak sequential write bandwidth with 4 KiB random writes and a sufficient number of threads (4).
• XFS, ext4, and the raw block device are comparable in write bandwidth.
• A larger block size means you need fewer threads to reach the peak sequential write bandwidth.

Read-only

• Achieving the peak sequential read bandwidth with 4 KiB random reads is not possible with up to 16 threads.
• Generally "more difficult" to achieve high read throughput with random reads (more difficult than random writes).
• 4 KiB random reads are slower (throughput) than 4 KiB random writes.

Mixed

• Highest achievable throughput is lower than both the peak read and peak write throughput of the SSD.
• In a 50/50 workload the read and write throughputs are the same. This is because they are limited by the slower of the two.
• Read throughput is the limiting factor. Read latency is higher than write latency.
• In an unconstrained workload (i.e., not 50/50), write throughput is generally higher than read throughput because writes are lower latency (which implies more writes can be issued in the same amount of time).

[geoffxy]

Closing this for now - we have enough conclusions for the P4510.