Tracking Issue: Automated Test/Support Coverage Reporting

[PhilippvK]

As discussed on Friday, we need an (automated) flow to track and report which kinds of instructions are currently being supported by Seal5. As the project is quite ambitious, I came up with a number of sub-tasks which can be implemented on individually.

Tasks

1. Properties of Instructions

   • [ ] Define relevant instruction properties
   • [ ] Extract instruction properties from metamodel
   • [ ] Write instruction properties to file

2. Status/Support of Instructions in Seal5 Flow

   • [ ] Extend pass metrics to emit names of failing/skipped instructions
   • [ ] Write status/supported patches to file

3. Tracking LLVM Test Results

   • [ ] Map test files to instructions
   • [ ] Parse test results
   • [ ] Store test results in metamodel
   • [ ] Write test results to file

4. Analysis and Grouping

   • [ ] Merge artifacts (properties, status, tests) in single table
   • [ ] Define useful filters/hierachy/…
   • [ ] Automatically combine similar instructions (same properties) into groups
   • [ ] Aggregate status/tests per group

5. Generation of Reports

   • [ ] Propose visualization methods (static/dynamic, table/tree/grid?)
   • [ ] Pick suitable visualization tool
   • [ ] Generate Reports
   • [ ] Publish reports

6. Automation (CI/CD)

   • [ ] Define triggers (weekly/push/tag/…)
   • [ ] Add workflows to refresh/release reports
   • [ ] GitHub Pages integration?

7. Miscellaneous

   • [ ] Allow combining reports from multiple Seal5 runs/flows
   • [ ] Increase number of available tests
   • [ ] Increase number of example instructions/extensions

[PhilippvK]

@thomasgoodfellow @dlr-jas @kebi-be @AndHager Feel free to add any comments or assign any tasks you would like to work on.

Tomorrow I can share some more notes!