Fault injection with bit flip

[ligurio]

something like bit flip Data degradation is the gradual corruption of computer data due to an accumulation of non-critical failures in a data storage device. The phenomenon is also known as data decay, data rot or bit rot.

Data degradation results from the gradual decay of storage media over the course of years or longer. Causes vary by medium:

Solid-state media, such as EPROMs, flash memory and other solid-state drives, store data using electrical charges, which can slowly leak away due to imperfect insulation. The chip itself is not affected by this, so reprogramming it approximately once per decade prevents decay. An undamaged copy of the master data is required for the reprogramming. Magnetic media, such as hard disk drives, floppy disks and magnetic tapes, may experience data decay as bits lose their magnetic orientation. Periodic refreshing by rewriting the data can alleviate this problem. In warm/humid conditions these media, especially those poorly protected against ambient air, are prone to the physical decomposition of the storage medium.[3][4]

see charybdefs 9

• https://github.com/scylladb/charybdefs/blob/master/cookbook/recipes.py

• https://github.com/scylladb/charybdefs/blob/master/tests/out_of_disk_space_test.py

• Empirical Measurements of Disk Failure Rates and Error Rates https://www.microsoft.com/en-us/research/publication/empirical-measurements-of-disk-failure-rates-and-error-rates/

• Bit Rot: How Hard Drives and SSDs Die Over Time https://www.howtogeek.com/660727/bit-rot-how-hard-drives-and-ssds-die-over-time/

  There’s a lot more to it than that, but this provides a basic idea of how the two storage types keep their data. Now let’s look at how they can lose it through bit rot. With hard drives, as mentioned above, saved bits can flip their magnetic polarity. If enough of them flip without being corrected, that can lead to bit rot. Solid-state drives, meanwhile, lose their data when the insulating layer degrades and the charged electrons leak out.

How long it takes to see bit rot in practice depends on a variety of issues. Hard drives have the potential to last with their data intact for decades even if powered down. SSDs, meanwhile, are said to lose their data within a few years in the same state. In fact, there are reports that, if they’re stored in an unusually hot location, the data on an SSD can be wiped out even faster.

• The Truth About SSD Data Retention https://www.anandtech.com/show/9248/the-truth-about-ssd-data-retention
• DRAM Errors in the Wild: A Large-Scale Field Study Report #1 mentions vendors declaring "Bit Error Rate of 10-12 for their memory modules", "a observed error rate is 4 orders of magnitude lower than expected". For memory related tasks, at a rate of 8 GBps this means a single bit flip may occur every minute (10-12 vendors BER) or once in two days (10-16 BER)
• https://indico.cern.ch/event/13797/contributions/1362288/attachments/115080/163419/Data_integrity_v3.pdf According to #2, there can be up to 25000-75000 one-bit FIT per Mbit (failures in time per billion hours), which is equal to 1-5 bit errors per hour for 8GB of RAM according to my napkin. Paper says the same: "mean correctable error rates of 2000–6000 per GB per year".
• Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing https://www.fiala.me/pubs/papers/sc12-redmpi.pdf The #3 report says, double bit flips "were deemed unlikely" but at ORNL's Cray XT5 they were observed "at a rate of one per day for 75,000+ DIMMs" even with ECC. And single-bit errors should be higher.
• Bitrot and atomic COWs: Inside “next-gen” filesystems https://web.archive.org/web/20150306225935/http://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/

• https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/index.html

In a recently initiated effort, Schwarz et al. [28] have started to gather failure data at the Internet Archive, which they plan to use to study disk failure rates and bit rot rates and how they are affected by different environmental parameters. In their preliminary results, they report ARR values of 2-6% and note that the Internet Archive does not seem to see significant infant mortality. Both observations are in agreement with our findings.

https://stackoverflow.com/questions/24181878/how-to-random-flip-binary-bit-of-char-in-c-c

[ligurio]

Yet another example of bit flip in production:

• Отъявленные баги и как их избежать на примере ClickHouse https://habr.com/ru/company/oleg-bunin/blog/497334/
• A malformed znode prevents ClickHouse from starting #780 https://github.com/ClickHouse/ClickHouse/issues/780

[ligurio]

See https://github.com/aybabtme/bitflip