Backend: Identify Common Issues — Modified Test Files

[frozenfrank]

This is a sub-issue of #266 that describes and defines one issue that can be solved independent of the others.

Overview

• Affects phases: all
• Root cause: Modified test files
• Prevents error: Test cases fail on the autograder when they pass locally. This can be caused by students:
  • accidentally allowing Intelli-J to refactor the test cases, by
  • having old versions of the test cases, OR
  • by actually changing the test files.
• Tip on detection:

  It looks like your test files have changed. This may have happened automatically if your IDE automatically refactored code while moving files. This could lead to the autograder giving different results than your local machine.

Detection Strategy

1. Hash their test files against two different version: one with the CRLF line endings, and one with the LF line endings.
2. Run git diff on the known test file and the student's file as demonstrated by this StackOverflow answer.

Option number 2 may be easier to implement since we can easily configure it to ignore whitespace differences. Additionally, we may be able to run it on entire directories of test objects in a single command instead of having to compare named files one-by-one.

Implementation Details

See the following details about ignoring whitespace details from git diff --help:

       --ignore-space-at-eol
           Ignore changes in whitespace at EOL.

       -b, --ignore-space-change
           Ignore changes in amount of whitespace. This ignores whitespace at line end, and considers all other sequences of one or more whitespace characters to be
           equivalent.

       -w, --ignore-all-space
           Ignore whitespace when comparing lines. This ignores differences even if one line has whitespace where the other line has none.

       --ignore-blank-lines
           Ignore changes whose lines are all blank.