goblint / analyzer

Static analysis framework for C
https://goblint.in.tum.de
MIT License
160 stars 72 forks source link

Add ARG tests #1470

Closed sim642 closed 1 month ago

sim642 commented 1 month ago

Like in #1462, I'm always afraid of doing any modifications in the ARG generation because it's very fragile and there have been no tests so far.

At some point I added https://github.com/goblint/analyzer/blob/bffc5e3cbb1f43bd5c9483a6545802cfba5d3d76/scripts/sv-comp/witness-isomorphism.py to semi-automatically check if two sets of GraphML witnesses are isomorphic. But that requires a set of reference witnesses (which we don't have in the repository) and the networkx Python library that implements graph isomorphism algorithms.

This PR adds an alternative means of fully automatic tests for ARGs. This is similar to the CFG tests: a GraphViz graph is rendered as ASCII using graph-easy and expected outputs are in the repository. The tests are very similar to Cram tests: if something changes, the tests will fail with a diff that can be promoted if intended. However, they aren't Cram tests but just dune expectation tests using rule generation.

The mechanism is somewhat complex (at least without dune 3.14), but nobody other than me will probably ever have to deal with these anyway (because they should just keep passing).

TODO