"Score for this problem" is incorrect

[je0k]

I think I have autolab working for our first homework in 08671-f15. (I still need to tweet the actual problems, but a prototype with three problems appears to work.)

The "bug" is that the last line of the output, which Autolab generates, is "Score for this problem: xx.x". The value (for xx.x) doesn't seem to be the sum of the scores. It appears to be the last score. I think this will be confusing to the students. Shouldn't this be the sum of the scores?

[jez]

I looked into this today, but I couldn't find an example of where it happened for your class, so maybe my answer is a bit off.

Generally the last line of the output should be a valid JSON string; this is how Autolab determines the final autograded results to assign points. If this was not the last line, it could have failed due to an error, or you might have found a bug.

As far as the issue of the wording of "Score for this problem," students using Autolab seem to be generally familiar with one assignment being broken into multiple "problems," each of which have their own sub-score, so I'm not sure that this is too big of a problem.

Let me know if I've overlooked something.

[je0k]

Hi Jake, We had some difficulties getting out grading program to work. We also had problems getting "regrade" to work, so we just submitted new versions of the homework -- so that's why there are so many test submissions for the 1st homework.

Anyway, the last assignment turned in (je0k@andrew.cmu.edu_11_Poem.java) seems to be graded correctly with a score of 23 out of 30. But the "Score for this problem" at the bottom shows the score to be "10.0"

Here's the output for that last submission (je0k@andrew.cmu.edu_11_Poem.java):

Autograder [Wed Jun 10 17:50:45 2015]: Received job 08671-f15_homework1_11_je0k@andrew.cmu.edu:495 Autograder [Wed Jun 10 17:50:55 2015]: Success: Autodriver returned normally

Autograder [Wed Jun 10 17:50:55 2015]: Here is the output from the autograder:

Autodriver: Job exited with status 0 tar -mxf autograde.tar java -version openjdk version "1.8.0_45" OpenJDK Runtime Environment (build 1.8.0_45-b13) OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode) java -cp bin grader.HW1Grader * Compiling file * javac Poem.java

* Running file * java Poem Hello World

* Checking Coding Conventions * Starting audit... /home/autograde/autolab/Poem.java:3: Missing a Javadoc comment. /home/autograde/autolab/Poem.java:3:1: Utility classes should not have a public or default constructor. /home/autograde/autolab/Poem.java:5: 'method def modifier' have incorrect indentation level 8, expected level should be 4. /home/autograde/autolab/Poem.java:5:1: File contains tab characters (this is the first instance). /home/autograde/autolab/Poem.java:5:9: Missing a Javadoc comment. /home/autograde/autolab/Poem.java:6: 'method def' child have incorrect indentation level 16, expected level should be 8. /home/autograde/autolab/Poem.java:8: 'method def rcurly' have incorrect indentation level 8, expected level should be 4. Audit done. Checkstyle ends with 7 errors.

{ "scores": { "file compiles": 10.0, "correct output": 10.0, "coding conventions": 3.0 } }

Score for this problem: 10.0

Graded by:

[dlbucci]

Each problem has a score associated with it and each score has feedback. When an autograder returns scores for multiple problems, all scores that are saved in the process save the autograder feedback as feedback. So when you view the autograder feedback for "file compiles" or "correct output", the score for those problems is 10, while "coding conventions" will be 3, but all three problems will have the autograder output saved.

As an aside, typing this out loud has made me realize why our scores table is over 20GB after one semester...

[je0k]

Thanks, Daniel. At least I understand the output now.

I'm surprised by the functionality. I think it would be confusing to the students. Certainly, all the staff didn't understand it (now, or in the past semesters, as far as I know). Is the "feedback" the output from running autograde-Makefile? Is there a way to get separate feedback for each problem? (I think the confusion is that we see the grading of every problem when we click to see results.)

Anyway, I guess this isn't a "bug". If I should take this up in a different forum, just let me know.

Thanks for all the help!

[dlbucci]

The feedback for each autograded problem is the output from autograde-Makefile, but instructors can also use the gradesheet to enter feedback for individual problems (such as style comments, or whatever) on the gradesheet. There's currently no way for an autograder to save different feedback for different problems though.

[je0k]

Thanks, Daniel and Jake.

On Fri, Jun 12, 2015 at 2:17 PM, Daniel Bucci notifications@github.com wrote:

Closed #522 https://github.com/autolab/Autolab/issues/522.

— Reply to this email directly or view it on GitHub https://github.com/autolab/Autolab/issues/522#event-329829390.