ls1intum / Ares

The Artemis Java Test Sandbox. A JUnit 5 Extension for Easy and Secure Artemis Java Testing
https://ls1intum.github.io/Ares/
MIT License
18 stars 7 forks source link

Introduce incremental structural Tests #377

Open pvlov opened 3 days ago

pvlov commented 3 days ago

Is your feature request related to a problem? Please describe.

Whenever you have an exercise which tasks the student to create lots of classes and may even include multiple levels of inheritance etc, you would usually use sequential testing to make this easier. This works perfectly fine, however this is most often quite annoying for the student since they first have to create all the classes and method stubs and so on. Ideally you should be able to test the structure incrementally for each class e.g. first test structure for class A then run the behaviour tests for class A.

Describe the solution you'd like I do not want to change the way sequential tests are run, they work exactly as they should. What I'd ideally want is to expand the existing {Method | Class | ... }TestProviders. The new method would then return a Map<String, DynamicTest>, that maps the standard test method names to the actual test, which then can be run whenever you need it. You could then run the structural test per class by just putting it in the respective @BeforeEach method.

Describe alternatives you've considered You can achieve the same by using the Dynamic{Method | Class |....} and then set boolean flags to then fail behavioural tests, but that doesn't really scale well.

Ares-Version you are using Most recent one.

I'm up for implementing this, but I would like your opinion and feedback if this is desired or if I'm maybe overlooking something.

pvlov commented 3 days ago

This would probably look something like this:

public Map<String, DynamicTest> generateTests() {
        Map<String, DynamicTest> tests = new HashMap<>();

    if (structureOracleJSON == null) {
        throw failure("The MethodTest test can only run if the structural oracle (test.json) is present. If you do not provide it, delete MethodTest.java!"); //$NON-NLS-1$
    }

    for (var i = 0; i < structureOracleJSON.length(); i++) {
        var expectedClassJSON = structureOracleJSON.getJSONObject(i);
        // Only test the classes that have methods defined in the structure oracle.
        if (expectedClassJSON.has(JSON_PROPERTY_CLASS) && expectedClassJSON.has(JSON_PROPERTY_METHODS)) {
            var expectedClassPropertiesJSON = expectedClassJSON.getJSONObject(JSON_PROPERTY_CLASS);
            var expectedClassName = expectedClassPropertiesJSON.getString(JSON_PROPERTY_NAME);
            var expectedPackageName = expectedClassPropertiesJSON.getString(JSON_PROPERTY_PACKAGE);
            var expectedClassStructure = new ExpectedClassStructure(expectedClassName, expectedPackageName, expectedClassJSON);

            var displayName = "testMethods[" + expectedClassName + "]";
            tests.put(displayName, dynamicTest(displayName, () -> testMethods(expectedClassStructure)));
        }
    }

    if (tests.isEmpty()) {
        throw failure("No tests for methods available in the structural oracle (test.json). Either provide attributes information or delete MethodTest.java!"); //$NON-NLS-1$
    }

    return tests;
}