Support custom static analysis

[pdawyndt]

TESTed is mainly focussed on dynamic analysis (= perform a unit test on the submitted code (function or method) or on execution of the entire code, and check the generated outputs with the corresponding expected outputs). However, many judges also support static analysis that only inspects the submitted source code without running it. Currently, TESTed only supports two built-in ways for static analysis:

• compilation: compilation errors are reported either globally or per unit
• linting: linting output is reported as source code annotations

Since there are many other forms of static analysis, we might add support for custom static analysis to TESTed. Here are some issues that need to be resolved to support custom static analysis in TESTed:

• perform custom static analysis globally (once for each submission), per unit (reported per unit) and/or per context (reported per context); there seems no value in performing custom static analysis below the context-level (individual testcase or individual test)
• language-independent vs language-specific static analysis: since the input is the source code of the submission, we could initially provide language-specific analysis; language-independent would require a generic API that needs to be implemented for each supported programming language and seems currently out-of-scope
• report results of static analysis in feedback: static analysis could be a simple binary check (e.g. function uses recursion) that could could be reported as a separate test, or have more elaborate output (software quality metrics, test coverage #339, full HTML report); the results of static analysis can also be reported as source code annotations (i.e. as an extension to the standard linter)
• support custom static analysis from DSL

[niknetniko]

Some use cases for this (that are currently done with the Python judge) are:

• controleren van type hints
• controleren van gebruik van specifieke variabele
• controleren of er ergens een dictionary gebruikt wordt
• controleren of een functie gebruikt wordt in een andere functie
• controleren of er geen lijst gebruikt wordt
• controleren of een list comprehension gebruikt wordt
• controleren of de math bibliotheek gebruikt wordt
• tellen hoeveel functies er gebruikt worden
• tellen hoeveel if/elif er gebruikt worden

[BTWS2]

Some more use cases:

• controleren hoeveel lussen gebruikt worden
• controleren of er een while-lus gebruikt wordt
• controleren of er een for-lus gebruikt wordt