cmi.core.lesson_status of "failed" status not being set

[paulstevendev]

When testing failing an assessment, the value of cmi.core.lesson_status is not being updated.

Windows 10 Chrome (latest) AT v0.11.1 Framework: v5.24.7

Create a course and add Assessment and Spoor extensions. Set completion criteria to pass assessment and create an assessment with one question. Select wrong answer and submit.

Expected behaviour

Selecting the wrong answer should result in failing the assessment and cmi.core.lesson_status being set to "failed"

Actual behaviour

There is no update to cmi.core.lesson_status

Screenshots (if you can)

[paulstevendev]

It would appear if the assessement has unlimited attempts then it is returning and not calling the tracking:complete

I note if I set the number of attempts on the assessment to 1, it does report the "failed" status

[oliverfoster]

As far as I understand, an LMS will ususally end the learner's session once a completion status is posted.

[paulstevendev]

Previously, if you failed an assessment and had unlimited attempts - it would still set the cmi.core.lesson_status to "failed" but you could still attempt the assessment again and pass it. When closing the course it would therefore report the lesson status as "passed". However if you did not attempt the assessment again, and closed the course, the LMS would record a lesson status of "failed". I had not read anywhere of any change on this front though we have only fairly recently upgraded to V5 from V3. If you had to set the number of attempts to 1 to ensure each failed attempt is recorded to the LMS then you are adding the requirement for the user to close the course and relaunch it just to have another attempt at the assessment. I am not saying this is necessarily wrong - just it has changed and I hadn't noticed any notification of this faily major change.

[oliverfoster]

It happened in May 2022 https://github.com/adaptlearning/adapt-contrib-core/commit/1f3907e59b82878a409568bf8c06c09d349b3cca

As a result of this issue https://github.com/adaptlearning/adapt_framework/issues/1448

[oliverfoster]

I see what you mean.

In your scenario the user is always completing the course in each session, regardless of whether they pass or fail. You want them to have unlimited attempts at the assessment during that session?

[paulstevendev]

Yes historically that is always how our Adapt assessment courses have worked. So if the user fails the course and closes the course down, next time it is lauched the LMS resets everything. Our LMS essentially resets everthing if the lesson status is "failed", "passed" or "completed" So the way this is now working would be a pain as would involve amending a lot of courses. If we had set a limit to the number of attempts on an assessment, this limit only applies for the duration of the session. We did encounter other LMS systems that would not reset things so setting a limit of say 3 attempts would mean once a user has failed 3 times they are unable to try again as on these LMS systems closing the course and relaunching did not reset things after a fail - which is a bit of a problem for statutory and mandatory content.

[paulstevendev]

Just to add, from an analytics perspective, being able to see the number of times a user has failed an assessment is very useful. If we wish to allow the user as many attempts as they want to try and pass the assessment then there will be no record of each of the failed attempts with the current implementation as far as I can understand.

[paulstevendev]

I created a course in Articulate Rise as a comparison and set it to unlimited attempts. I failed the quiz (assessment) and closed down the course and it reported "failed" to the LMS despite having unlimited attempts so it would appear how it used to work in Adapt is how it works in Rise.

So I would appreciate some further discussion on this functionality in Adapt to see if there was any way to have an option to make it work the way it used to work.

[oliverfoster]

I shouldn't be difficult. Just need a flag to make it work as it did.

[paulstevendev]

That would be awesome if you can action this. I suspect the wording for that flag could be tricky to come up with.

[danielghost]

@paulstevendev - The "infinite" attempts and completion logic was noted in the updtaed readme https://github.com/adaptlearning/adapt-contrib-assessment/pull/154/commits/87caf781c686faaf2a40b3dc69b9b14f67d90b61, but can see how this could have been missed.

I would query the design logic for using unlimited assessment attempts when the LMS has been configured to reset the progress once the browser session is closed. In this case I would probably restrict the user to 1 attempt, as it seems odd to allow multiple attempts within the session when it doesn't make any difference to whether the course gets reset when they return. Using 1 attempt means the user journey is consistent and the completion status would also be sent every time.

From an analytics perpsective to see the history of assessment attempts, the score still gets sent for every attempt, so assuming appropriate reports were available to look at all recorded scores, this could be used to see that history. I can see the argument for also wanting to see a history of recorded completion statuses. This is now configurable in https://github.com/adaptlearning/adapt-contrib-core/pull/316/commits/53827f77f493aab6cad7409885a575ad15258026.

[paulstevendev]

@danielghost - thank you for your explanation. When you say the LMS resets the user progress when the browser session is closed, our LMS as far as I am aware only resets everything in the scenario where the lesson status is passed, failed or complete. Obviously you would not want to reset a users progress if they are partially through completing a course - and surely the reason for having suspend data is to allow a user to resume. So in our circumstances if a user fails the assessment and they only had one attempt, they would not be able to attempt it again as the LMS would not reset the course.

That is a good point abou the analytics and the score being sufficient to determine a user has failed though it would require the LMS to have a handle on the pass mark for every session and as this is determined by the course itself.

Anyway that is great news this is now configuable.

[danielghost]

@paulstevendev - Sorry if this wasn't clear in my previous comment, but I understand the LMS was only resetting on a "passed"; "failed"; "completed" . As you say, it would make no sense for it to reset when incomplete. tracking:complete should have been triggered if the user only had 1 assessment attempt as the completionData.assessment?.canRetry condition would be bypassed, which isn't the case with "infinite" attempts prior to this fix.

It doesn't matter now that you can hopefully configure it as needed, but my point was more that it seems odd to allow a user to have more than 1 attempt knowing that the LMS will reset the progress when they exit, as the resetting beheviour happens regardless of whether they passed or failed anyway. I get that you may still want them to have the ability to achieve a "passed" rather than "failed", but my main concern is that in your case, they can only do that when they are still within the existing session. If a user had failed and then needed to close the course to go to a meeting etc. they are now reset when they return (and may have to work through all the content again before they can go back into the assessment, depending on the design) which is a different user journey to those who happened to retry within the session.

Just a note to say that if you were using SCORM 2004, where the default behaviour is for an LMS to reset the cmi.suspend_data for completed users, this can be overriden via "_exitStateIfComplete": "suspend".

[oliverfoster]

As far as I understand: The lms isn't resetting the course, it's starting a new course attempt, the running session is the last attempt, until the lms finished is closed. When executed for a second time, the course will be running in the second session.

[paul-mediakitchen]

@danielghost - thanks for the further clarification. I think I understand what you are saying with regards the tracking:complete I will do some tests on Moodle to try and test out what you are saying but I am really pleased you have made the update as that will ensure it works as it had previously worked.

[paul-mediakitchen]

I did some tests on Moodle with the version that only submits a "failed" status if it is the last attempt. If I set the Attempts Management setting for Force New Attempt to No then the user cannot retry the assessment again once they have failed - not even if they close the course down and relaunch it. A lot of our Adapt is hosted on a LMS that has such a setting. So having it work with unlimited attempts works well for us for this LMS. I am not sure of the reasoning for this LMS having this setting. On our own LMS however we do reset the course if the status is failed, passed or complete so it would not be a problem.