Add % of target methods that are compilable as a CI check

njit-jerse / specimin

SPECIfication MINimizer. A different kind of slicer for Java.

MIT License

3 stars 5 forks source link

Add % of target methods that are compilable as a CI check #292

Closed kelloggm closed 3 weeks ago

kelloggm commented 4 months ago

This directory has a set of useful scripts for running Specimin on every method in a project and reporting statistics about the % of them that can be compiled: https://github.com/njit-jerse/specimin-evaluation/tree/main/ashe_scripts

We should add a CI target that uses these scripts to run Specimin on a few interesting (to us) projects and fails CI if the % of compilable output is not monotonically-increasing.

jonathan-m-phillips commented 4 months ago

To be clear, in order to run the scripts with ASHE, we will need to point to a config.properties file somehow for Checker Framework and Specimin. ASHE is expecting to find those values from config.properties.