Give users ability to change disk behavior in Simulator

[mpilman]

Disks in the simulator are currently implemented in a very simple way: the simulator will make a blocking request and then delay for a random amount of time. This is good enough for most tests, but it is inflexible if one wants to test how FDB behaves. Furthermore it is currently not possible to simulate failing disks.

The problem here is that the test dimensions are getting huge. Optimally we would like to test the following:

• Different kind of disks (like SSD, HDD, EBS, GCP Block Storage, Azure Storage disks, NVM...)
• Different performance behaviors:
  • Running out of IOPS (so stop executing operations until the next second starts). This would be interesting to simulate Cloud provider's disks like EBS.
  • Larger differences in latency (for network disks) and higher latencies in general.
• Different kind of disk failures: There should be a low probability that a disk fails. This is hard to implement as too many disk failures (or disk failures on clusters without replication) can make a test fail legitimately. However, this is what I would like to simulate in this area:
  • Disk starts throwing errors for read/write operations.
  • Some operations return Never - simulating a blocking operation.
  • Read requests start reading garbage instead of the actual data.
  • Write requests write garbage.

[sfc-gh-amotivala]

We should definitely simulate the Azure Managed disk behavior where each second is spilt into 20 sub-interval for IO SLAs.

[xumengpanda]

I guess Azure uses the idea in MSR's paper: End-to-end Performance Isolation Through Virtual Datacenters [1].

One question: The SLA related regulation usually smooth the burst request (e.g., burst IO requests) by delaying the request. This will definitely affect performance. But will it affect correctness? (I guess simulation can still catch the OOM bug at the event of slow request.)

[1] https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-angel.pdf