Provide percent correct for parsons problems

[bnmnetp]

As part of the consistency effort for components I'm looking for a simple algorithm for computing a percent correct measure for parsons problems.

Calculation Method

I'm proposing the following three components with attached weights

1. percentage of number of lines. (num lines in answer / num lines in solution) - 20%
2. percentage of blocks in correct order - 40%
3. percentage of blocks indented correctly - 40%

Logging

It is easy enough to keep backward compatibility and simply add a new percent key to the json that gets sent to the server. I don't see any reason to add more stuff to parse to the poor act field.

For autograder

We can add a percent column to each of the xxxx_answers tables. if the grading method is % correct then the auto-grader can

Feedback? @barbarer @bhoffman0 @pearcej @bjones1

[pearcej]

Something like this method makes good sense to me.

Of course, although some languages do not REQUIRE indenting, I expect most instructors still teach it for visual recognition of logical blocks. The challenge comes with those pesky brackets in languages like Java and C++…

I would love to hear thoughts about this…

Jan

“Resilience is accepting your new reality, even if it's less good than the one you had before. You can fight it, you can do nothing but scream about what you've lost, or you can accept that and try to put together something that's good.” ― Elizabeth Edwards

---

Dr. Jan Pearce | Preferred pronouns: she, her, and hers Chair, Computer Science Department, (859) 985-3569 Professor, Computer and Information Science Address: Berea College | Berea, KY 40404 Web: www.berea.edu/csc/faculty-and-staff/dr-jan-pearce/http://www.berea.edu/csc/faculty-and-staff/dr-jan-pearce/ Calendar: bit.ly/pearcej-calendar</bit.ly/pearcej-calendar> | Classes: pearcej.github.iohttp://pearcej.github.io/

From: Bradley Miller notifications@github.com Sent: Tuesday, November 10, 2020 2:39 PM To: RunestoneInteractive/RunestoneComponents RunestoneComponents@noreply.github.com Cc: Jan Pearce pearcej@berea.edu; Mention mention@noreply.github.com Subject: [RunestoneInteractive/RunestoneComponents] Provide percent correct for parsons problems (#1097)

[EXTERNAL SENDER]

As part of the consistency effort for components I'm looking for a simple algorithm for computing a percent correct measure for parsons problems.

Calculation Method

I'm proposing the following three components with attached weights

1. percentage of number of lines. (num lines in answer / num lines in solution) - 20%
2. percentage of blocks in correct order - 40%
3. percentage of blocks indented correctly - 40%

Logging

It is easy enough to keep backward compatibility and simply add a new percent key to the json that gets sent to the server. I don't see any reason to add more stuff to parse to the poor act field.

For autograder

We can add a percent column to each of the xxxx_answers tables. if the grading method is % correct then the auto-grader can

Feedback? @barbarerhttps://github.com/barbarer @bhoffman0https://github.com/bhoffman0 @pearcejhttps://github.com/pearcej @bjones1https://github.com/bjones1

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/RunestoneInteractive/RunestoneComponents/issues/1097, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAMISWLXMZJMP3NSXCJ77CLSPGJD7ANCNFSM4TRCNJQQ.

[barbarer]

I would rather give people the highest points possible. I first check if they have the same number of blocks as the correct solution and if so I usually give 0.5 point if they have the right block in the correct place, but the indentation is wrong and 1 point if the block is in the correct place and it is indented correctly.

I also grade by comparing the first block in their solution to the correct solution and continue grading while the blocks match or if they used a distractor for the correct block until I find the first incorrect block (0.5 for correct block with wrong indentation and 1 point for correct block with the correct indentation). If I haven't checked their complete solution I also start at the end of their solution and grade again until I find an incorrect code block or reach the same index. The score is the points / (max of total blocks in the correct solution or the number of blocks in thier solution).

I gave them the max possible points from using these two methods.

[bnmnetp]

Yikes this is the slippery slope of writing an LMS people have different ideas on how to grade, how generous to be with partial credit, etc. etc.

At the same time we have to remember that Parsons can be used for so many things, as Jan points out, some of which do not rely at all on indentation.

I'm worried that we create a really complex system that people will not understand or agree with.

[barbarer]

When you create a Parsons problem you can decide if the student must indent or not. If you require indentation I would think that it should be graded as well. It might be best to have several grading options and let the instructor decide.

Dr. Barbara Ericson Assistant Professor, School of Information University of Michigan

On Tue, Nov 10, 2020 at 6:13 PM Bradley Miller notifications@github.com wrote:

Yikes this is the slippery slope of writing an LMS people have different ideas on how to grade, how generous to be with partial credit, etc. etc.

At the same time we have to remember that Parsons can be used for so many things, as Jan points out, some of which do not rely at all on indentation.

I'm worried that we create a really complex system that people will not understand or agree with.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/RunestoneInteractive/RunestoneComponents/issues/1097#issuecomment-725023658, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKOZ7M7NYLB6WSYUJ3MP3MLSPHCI7ANCNFSM4TRCNJQQ .

[bnmnetp]

My feelings on multiple grading options are that it just makes the system more complicated. This is why all LMSs are hard so hard to use. Just add an option so that everyone can do things however they want. Ugh! The result is that few people can figure out how to do even the basics.

From where we stand today allowing the Parsons problems to be graded by % is a step forward. Given the limited number of contributors, there are a lot more important things to work on than adding the UI for more grading options for this one question type and allowing teachers to choose. If someone really doesn't like the grading algorithm they can always manually override it.

My main motivation for doing this is so we have consistency across the components. Everything should be able to be graded on a percent correct. Even a True/False question should be gradable on a percent correct even if the only two possible percentages for that question are 0 and 100.

I'll take another shot at an algorithm that looks both top down and bottom up and takes into account whether indentation is required. It seems overly generous to give the 100% of the points for indentation when it is not really part of the problem.

[bnmnetp]

OK, humor me for a minute. Here are some samples: If you were grading these MANUALLY and giving partial credit how many points would you award (out of 10) for each:

This one gets 10 since it is correct.

1. 
2. 
3. 
4. 

[barbarer]

1. 5/5 2 3/5 3 4.5 / 5 4 4/ 5 5 3/5

This is the easy case when the number of lines in the user's solution is the same as the number of lines in the correct solution. It is harder to grade the case when the student's solution has more or less lines than the correct. That is when I use the other algorithm.

Dr. Barbara Ericson Assistant Professor, School of Information University of Michigan

On Wed, Nov 11, 2020 at 2:17 PM Bradley Miller notifications@github.com wrote:

OK, humor me for a minute. Here are some samples: If you were grading these MANUALLY and giving partial credit how many points would you award (out of 10) for each:

This one gets 10 since it is correct. [image: Screen Shot 2020-11-11 at 1 13 29 PM] https://user-images.githubusercontent.com/51115/98854328-07012100-2420-11eb-8859-b1ec18605373.png

1.

[image: Screen Shot 2020-11-11 at 1 10 16 PM] https://user-images.githubusercontent.com/51115/98854374-16806a00-2420-11eb-8f56-e01b2bdfeaee.png

1.

[image: Screen Shot 2020-11-11 at 1 09 25 PM] https://user-images.githubusercontent.com/51115/98854394-1da77800-2420-11eb-9bdd-fb2426271921.png

1.

[image: Screen Shot 2020-11-11 at 1 08 26 PM] https://user-images.githubusercontent.com/51115/98854418-26984980-2420-11eb-863f-dc7157932b97.png

1.

[image: Screen Shot 2020-11-11 at 1 06 27 PM] https://user-images.githubusercontent.com/51115/98854455-31eb7500-2420-11eb-81d0-8cc834ec3810.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/RunestoneInteractive/RunestoneComponents/issues/1097#issuecomment-725610852, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKOZ7M36XE4MJUWTEIWVCH3SPLPN5ANCNFSM4TRCNJQQ .