rems-project / cerberus

Cerberus C semantics
https://www.cl.cam.ac.uk/~pes20/cerberus/
Other
48 stars 25 forks source link

[CN] Caputre all output (not just return codes) for CI in a convenient to update way #412

Open dc-mak opened 1 month ago

dc-mak commented 1 month ago

We don't want unnecessary duplication of C files, since any changes to syntax or semantics would need to be duplicated across multiple tests.

Perhaps if/when CN is a library, we can use some expect tests (including for tutorial. This would also be useful for checking the behaviour/output of tutorials once those are integrated.

A small example I made a while back is here: https://github.com/dc-mak/NumLin/tree/master/test

dc-mak commented 4 weeks ago

Update

With the additions of

I propose that, in addition to return codes, for a given files we have the ability to specify the expected output in a filename.expected if we want to to use a script like this or use expect tests as stated above. A key feature of both is that tests can be auto-promoted when behaviour changes en masse