One of the best-tested pieces of code in the world is called SQLite, the database mentioned earlier that’s probably on your smartphone. It was developed by D. Richard Hipp, who’s been working on it for 15 years. It’s totally open, totally free, and has 33,402 tests. It’s one of the most widely used pieces of software and one of the most respected.
Filing an issue regarding the number of tests in SQLite feels like nitpicking, but the test count here is incorrect by more than an order of magnitude.
SQLite uses a number of different unit test strategies. The one available alongside the source is the TCL test suite. Within those tests, the veryquick suite represents the highest-impact tests that can run in a (relatively) short amount of time. As of ~3.8.10.2, it contains 205,754 test cases.
The slower tests involve testing things like verifying database integrity in the face of inopportune power loss and memory allocation failures (such as testing what happens when malloc returns NULL for literally any and all calls to malloc)—such tests can be quantified by the number of cases, but are far more meaningful for their completeness of coverage. As of 3.8.10, SQLite is also under constant scrutiny from the afl fuzzer.
Filing an issue regarding the number of tests in SQLite feels like nitpicking, but the test count here is incorrect by more than an order of magnitude.
SQLite uses a number of different unit test strategies. The one available alongside the source is the TCL test suite. Within those tests, the
veryquicksuite represents the highest-impact tests that can run in a (relatively) short amount of time. As of ~3.8.10.2, it contains 205,754 test cases.The slower tests involve testing things like verifying database integrity in the face of inopportune power loss and memory allocation failures (such as testing what happens when malloc returns NULL for literally any and all calls to malloc)—such tests can be quantified by the number of cases, but are far more meaningful for their completeness of coverage. As of 3.8.10, SQLite is also under constant scrutiny from the afl fuzzer.
And yet, testing does not catch everything. There's no substitute for formal proofs when it comes to guaranteeing behaviour.