Verification if submission programming interfaces are implemented

[pdawyndt]

Reasoning: Dynamic testing hooks into the required programming interfaces implemented by the submission. To enable this, the task description must specify what programming interfaces must be implemented by submissions. Test suites could specify what programming interfaces must be implemented (for the whole submission or for a single test unit), so that TESTed could verify (static testing) if the necessary interfaces were implemented prior to starting dynamic testing.

Advantages: No CPU cycles are waisted on dynamic testing for a unit whose programming interfaces have not been implemented by the submission. More specific feedback can be reported to students.

[pdawyndt]

Language-specific modules must implement introspection to detect if the following programming interfaces have been implemented:

• for submission that must be compiled and executed as standalone executables:
  • standalone compilation of submission
• for submissions whose main function is called:
  • "main" function
• for unit testing of submissions:
  • global variables
  • functions
  • classes
  • methods
  • properties
  • parameters (functions, constructors, methods)

[pdawyndt]

Clean format for specifying programming interfaces needed in test suites. TESTed could provide helper subcommand to derive test suite specification of programming interfaces from sample solution (e.g. sample Python implementation usign type hints).