New 'judge' optional property on some interactivity pages - issues/suggestions

FayCross commented 2 months ago

Some issues with the way this works on the different pages:

Categories:

Retested 25/10 - works well now

[x] the results page no longer works after a categories page has been viewed.
[x] if you turn judge off you still get the 'you scored x out of y' text and the labels are still marked with a tick or a cross

MCQ

Retested 25/10 - unchecked issues below are still there

[x] some existing projects will now have MCQ pages that behave differently. Previously, if you set up an MCQ with no answers marked as correct then the 'your answer is incorrect' text did not show on the screen after submitting an answer. Now, unless you have added the judged optional property and unchecked it, the incorrect feedback will always show for these activities. There needs to be backwards compatibility for this to work as lots of people do this.
[x] there is already a property 'mark answer' within the feedback property group that I think should be used instead of the separate 'judge' property. This already controls whether the 'your answer is correct/incorrect' text shows (if at least one answer is marked as correct) so it's confusing to have another option that does pretty much the same thing. Ideally, I think we should assume the page is not judged if either there is no answer marked as correct, or the mark answer box is unchecked.
[ ] Results page shows score of 100% even if no answer submitted (regardless of whether mark answer is on or off)

Answer X of Y

Retested 25/10 - unchecked issues below are still there

[x] Even if answers are set to not be judged, the text fields are all ticked after submitting answers (even empty ones). Ideally there would be no ticks shown on the page.
[ ] If there are 10 boxes and only 5 are completed should the mark on results pages still be 100%? Would 50% be more appropriate as only 50% of the boxes were completed?
[ ] If only 10 out of 5 boxes are completed, the results page shows empty rows on the results page in the interactivity results section.
[ ] After submitting you can't edit your answers - even if you only completed 5 of the 10 boxes before clicking submit
[x] The attempts optional property is not relevant when judging is off - should this option be hidden when judge box is checked?
[x] If you only complete 5 out of 10 boxes and submit, you still get the tick and 'you have completed the activity' text.
[ ] When marking is off, you still get the 'not all of the answers are correct' text shown

Dialogue

Retested 25/10 - unchecked issues below are still there

[ ] When marking is off, you can submit blank answers and you still get the score as 100% on the results page
[x] Not sure answers should be ticked when judging is off - just move on to the next question?
[ ] When marking is on, if you submit some blank answers, as well as some correct answers, the score is always shown as 0%, even though the results table below shows rows with ticks in

Model Answer

Retested 25/10 - added extra notes below

I don't see the point of there being a judge property on this page. Previously, as long as you submit an answer then the results page shows a tick and a score of 100%. The only difference when judge is off seems to be that you can submit an empty text field and still get the 100% - this seems odd as I think to get a score then you should be expected to submit an answer
Note after 25/10: this seems to have been changed so you can no longer submit a blank answer when judge is off. The results page still shows the marks as 100% and the page as completed even if no answer submitted though. I still don't understand the benefit of being able to flag this page as not marked as this activity is never marked - there is no correct or incorrect answer and whatever students submit is already shown on the results page.

Sortable Grid

Retested 25/10 - unchecked issues below are still there

[x] breaks the results page after sortable grid page has been viewed
[x] labels are ticked after submitting when answers are not supposed to be judged. 'Your answers have been submitted' text would be more appropriate.
[ ] Results page is not showing all the data in the 'your answer' column correctly (regardless of whether marking is on or off). The top two rows show in the correct format (e.g. Iron -> [2,2]) but below that the info is shown without the label text (e.g. -> [4,2]). This means that even when labels are dragged correctly they are showed with a cross in the details table as the text in the your answer column does not match the text in the correct answer column. The overall score for the page at the top is correct but when you look at the details below it looks like you may have got all labels placed incorrectly

Media Lesson

Retested 25/10 - unchecked issues below are still there

[x] if all answer options are set to false then you get 'your answer is correct' text. In 3.12 you do not get that kind of feedback unless at least 1 answer is marked as correct
[x] when judge is off you still get the your answer is correct/incorrect text on the page which implies that the answer is still being judged
[x] when judge is off and the MCQ is a drop down menu the answer is still judged. If you get the answer wrong then the score shown on the results page is 0%
[ ] Unlike the other pages, if mark answer is off then the score is shown as 0% on the results page - even if you have submitted answers to all the questions
[x] The mark answer optional property is unchecked when first added. For consistency with other pages it should be checked when first added and then you can uncheck to stop marking.

Gap Fill

Retested 25/10 - unchecked issues below are still there

[x] Inge's release notes suggest that the 'fill in the blank' page also has the judge optional property. I can't see if on the gap fill page though - not sure which page in the fill in the blank page if not gap fill - NOTE: I have worked this out and it is only available when fill in the blank activity.
[ ] When marking is off you can submit partially completed activity and you get the 'you have completed this exercise' text. You can then no longer go back to complete the empty blanks
[ ] When the activity is only partially completed you get a score of 100% - is this right?

General comments on this functionality:

Retested 25/10 - unchecked issues below are still there

[x] What actually is this new functionality trying to achieve? I took this functionality as allowing students to express an opinion but on lots of the pages there are still ticks and 'you scored x out of y' text. If it's just opinion then it should not just be the results page that has changed functionality but also the scoring info on the pages themselves should be removed.
[x] Using the wording 'mark answer' instead of 'judge' would bring it more in line with wording we use elsewhere and be clearer. I don't think just 'judge' is very clear. If staying with judge wording then the property name needs a capital letter at beginning (on some pages it's all lower case) and the tooltip also needs to have capital letter & perhaps be reworded to make its function clearer.
[ ] When pages aren't judged, I think it would make more sense for no score to be reported on results page. So details of answer given is shown but score should just be a dash rather than 0% or 100%. This would solve lots of the issues I've mentioned above where partially completed exercises are still marked as having 100% score

FayCross commented 2 months ago

@torinfo I've started retesting this after recent changes and have added checkboxes to the points above that I've looked at (some seem to have been fixed/dealt with and some haven't). I'm going to stop testing this now though as I'm not sure I get this functionality at all. When I first tested, when judging was off all the pages showed info on results page but always scored at 100% (so they could be opinion style questions). Now none of the pages I've tested show any detailed info on the results page when judge is off. I thought this functionality was so data was still tracked but did not require answers to match a correct answer. Nothing seems to be tracked any more as far as I can see. Perhaps I am just misunderstanding but will hold off carrying on testing for now.

JMarco27 commented 1 month ago

@FayCross @torinfo Am I right in thinking that the idea of "Judge off" is that you use it when a) you are asking the learner's opinion, about something b) there is no "correct" answer c) you may want narrative feedback to display on completion. The only time I have wanted to use this is in a kind of prioritising task, where I've used Sortable Grid with one column, when learners are asked to rank what is important to them in a job -https://languagehub.uk/xerte/play.php?template_id=249#page1. I'd suggest when "Judge off" is selected it doesn't feed into the Results Page. However, am I also right in thinking that Results Page automatically includes information on performance on all "scorable" pages? Don't we need a wider discussion about marked/unmarked tasks, the purpose(s) of Results Page, and the implications of including it in an LO?

torinfo commented 4 weeks ago

Hi Jonathan,

You are correct about the rationale, but also to get the answers in the results page and xAPI. So, if it is not desirable to have the answer in the results page, we should make the results page more configurable.

FayCross commented 1 week ago

@torinfo again, this is much better. I've added extra notes above to the issues that are still there though. Most of them are because I think it's very confusing for unmarked activities to be given 100% score even if the activity was not completed. There are several pages where you can partially submit your answers, the results page will give you 100% and you are unable to edit your answers or add answers to parts of the activity that you left unanswered.

thexerteproject / xerteonlinetoolkits