pnevyk / gryf

Graph data structure library aspiring to be convenient, versatile, correct and performant.
MIT License
69 stars 1 forks source link

Beginning the correctness journey #46

Closed pnevyk closed 1 year ago

pnevyk commented 1 year ago

This PR establishes an infrastructure for correctness testing using fuzzing and property-based testing. There are two main areas:

Graph storages

For graph storages, we will mainly employ fuzzing with stateful modeling of a graph behavior. For that, we have a MutOp enum which represents common graph manipulating operations. The test input is a sequence of such operations, either completely random or guided via MutOpsSeq. The operations are then applied on a graph under test. There are three levels of such testing:

  1. The graph does not panic while doing an operation.
  2. The graph is in a consistent state after each operation. For this, there is the check_consistency utility function.
  3. The graph has the same behavior as a model of a graph when applying an operation. For this, there are the Model type and check_potential_isomorphism function.

The Model type is an implementation of graph with the goal of being simple so it is easier to reason about its correctness. Moreover, it provides ways how to adjust the operations sequence according to specific needs using the ModelParams type and check method. The model itself can be tested on levels 1 and 2.

I believe it will be better to check consistency and model-conformance after every operation applied to reveal problems as soon as they occur.

The main reason for using stateful testing with sequence of operations is that it allows to assert validity of the graph also after removing operations, which are usually the hardest to implement correctly.

Algorithms

For algorithms, we will mainly employ property-based testing with fast and customizable generation of the input graphs. For that, we will utilize proptest crate with custom GraphStrategy. The strategy is able to generate a graph into provided type if it implements standard graph manipulation traits. It is customizable via StrategyParams and thus is able to generate constrained graphs to reduce the number of cases when an algorithm would unnecessarily fail on a graph structure it does not support.

The graph algorithms outputs are relatively good for property-based testing. One can think of a few rules that apply to an output and often lemmas from the literature come handy. At the very least, we will usually have more than one algorithm for a problem, so that "oracle" test can be used to compare the results from different algorithms.

pnevyk commented 1 year ago

Of course, first attempts to use fuzzing on currently implemented storages revealed bugs even in very simple cases.

For example, undirected AdjList panics for

  1. Add vertex
  2. Add self-loop edge on that vertex
  3. Clear edges
  4. Try to remove self-loop edge on the vertex

After adding check for consistency, for directed AdjList we get even shorter sequence:

  1. Add vertex
  2. Add self-loop edge on that vertex
  3. Clear edges

Other storages ended up very similarly.

No doubt that this effort will help tremendously.