Processes freeze for random durations (done in #21)
"For a while, there was an informal competition within the engineering team to design failures that found the toughest bugs and issues the most easily. After a period of one-upsmanship, the reigning champion is called “swizzle-clogging”. To swizzle-clog, you first pick a random subset of nodes in the cluster. Then, you “clog” (stop) each of their network connections one by one over a few seconds. Finally, you unclog them in a random order, again one by one, until they are all up. This pattern seems to be particularly good at finding deep issues that only happen in the rarest real-world cases." https://apple.github.io/foundationdb/testing.html
FoundationDB nemeses:
Incomplete writes to disk(see #75)Corrupted writes to disk(see #75)Disk drive runs out of space(see #75)Single node in a cluster terminates(done in #16)Network partitions between nodes(done in #17)Processes freeze for random durations(done in #21)"For a while, there was an informal competition within the engineering team to design failures that found the toughest bugs and issues the most easily. After a period of one-upsmanship, the reigning champion is called “swizzle-clogging”. To swizzle-clog, you first pick a random subset of nodes in the cluster. Then, you “clog” (stop) each of their network connections one by one over a few seconds. Finally, you unclog them in a random order, again one by one, until they are all up. This pattern seems to be particularly good at finding deep issues that only happen in the rarest real-world cases." https://apple.github.io/foundationdb/testing.html