Backend: Identify Common Issues — Incorrect Test location

[frozenfrank]

This is a sub-issue of #266 that describes and defines one issue that can be solved independent of the others.

Overview

• Affects phases: all
• Prevents error: Autograder fails to find and evaluate their tests even though the student has written custom tests
• Tip on detection:

  Make sure files are in the exactly correct location. Make sure the java package is marked as the Test Sources Root. Make sure that the import statement shows the right package name.

Detection Strategy

Specifically detect if the provided package exists, and that the files inside have the correct package statement at the top. List the java test files that we did find to help point out that it should be in a different location.

Implementation Details

Define a list of all the packages that should contain test files for each phase. Specifically check if each File exists at that location (they'll be directories). If any of the required test packages are missing, we'll report this to the user.

Inside of each file, the package statement at the top can sometimes be messed up depending on how students move the files around during development. Also scan each of the files for the package statement. If it doesn't match the expected package statements, report it to the user including the package statement we found. This could be done with a regex of the form /^package .*;/

[Fiwafoofa]

@frozenfrank Just to clarify my understanding, when you say to verify that the package statement is correct given a file, what is the significance of this, i.e., what new thing does it contribute as Intellij (I think) should already verify if files have the wrong package statement. Adding this could be redundant and add some additional time to grading code as it would need to open each of the files and scan them for their package statement and might be worth excluding (although I already coded it up, so it wouldn't be a problem if no one cares about the little bit of extra time).

(Or am I misunderstanding the purpose of this feature?)

[19mdavenport]

On the topic, it might be useful to check for non-test starter code like the api test page files