Separate "Tests after deadline" from "secret tests"

[cod-eric]

Is your feature request related to a problem?

To my understanding, there are three types of tests:

1. test that are executed before a submission deadline, that give precise feedback on which issue caused it to fail (i.e. whether it was a wrong result, an exception, etc)
2. tests that are executed before a submission deadline, that give no feedback on which issue caused it to fail before the deadline has passed
3. tests that are only executed after a submission deadline, that then give precise feedback once they have been executed

Currently, 2 and 3 are a bit hard to distinguish on Artemis. Most courses I know achieve test type 2 by using secret tests that always fail before the deadline, which causes them to appear red in the percentage view after each push.

However, as long as the course owner does not clearly annotate which tests are actually failing before the deadline and which ones are simply not yet to be revealed, it is quite hard for students to distinguish these two.

Describe the solution you'd like

Therefore, I'd like to propose a distinction between these 3 types. Maybe tests marked as "executes after deadline" just show grayed out before the deadline, with the question mark icon currently used when a test hasn't run so far, making it clear to the student that these tests do not show an error in their code (yet)?

Describe alternatives you've considered

No response

Additional context

No response

[b-fein]

Internally the test cases are split as described in https://docs.artemis.cit.tum.de/user/exercises/programming/#configure-grading under point Visibility.

Right now the visibility of pass/fail and the test case feedback are directly tied to each other. I.e. either the student can see pass/fail and the test case feedback, or the student cannot see either.

If I understand you correctly: What you propose would be to split the ‘After Due Date’ case into two separate subcategories

• neither show pass/fail nor feedback before due date, both afterwards (current implementation),
• only show pass/fail information before due date, but additional feedback message after due date?

[cod-eric]

Yes, that is my proposal! I think it'd make it less confusing for students to know whether they actually fail a test or it is just not executed/graded yet.