ls1intum / Artemis

Artemis - Interactive Learning with Automated Feedback
https://docs.artemis.cit.tum.de
MIT License
502 stars 291 forks source link

Separate "Tests after deadline" from "secret tests" #6681

Open cod-eric opened 1 year ago

cod-eric commented 1 year ago

Is your feature request related to a problem?

To my understanding, there are three types of tests:

  1. test that are executed before a submission deadline, that give precise feedback on which issue caused it to fail (i.e. whether it was a wrong result, an exception, etc)
  2. tests that are executed before a submission deadline, that give no feedback on which issue caused it to fail before the deadline has passed
  3. tests that are only executed after a submission deadline, that then give precise feedback once they have been executed

Currently, 2 and 3 are a bit hard to distinguish on Artemis. Most courses I know achieve test type 2 by using secret tests that always fail before the deadline, which causes them to appear red in the percentage view after each push.

However, as long as the course owner does not clearly annotate which tests are actually failing before the deadline and which ones are simply not yet to be revealed, it is quite hard for students to distinguish these two.

Describe the solution you'd like

Therefore, I'd like to propose a distinction between these 3 types. Maybe tests marked as "executes after deadline" just show grayed out before the deadline, with the question mark icon currently used when a test hasn't run so far, making it clear to the student that these tests do not show an error in their code (yet)?

Describe alternatives you've considered

No response

Additional context

No response

b-fein commented 1 year ago

Internally the test cases are split as described in https://docs.artemis.cit.tum.de/user/exercises/programming/#configure-grading under point Visibility.

Right now the visibility of pass/fail and the test case feedback are directly tied to each other. I.e. either the student can see pass/fail and the test case feedback, or the student cannot see either.

If I understand you correctly: What you propose would be to split the ‘After Due Date’ case into two separate subcategories

cod-eric commented 1 year ago

Yes, that is my proposal! I think it'd make it less confusing for students to know whether they actually fail a test or it is just not executed/graded yet.