Grading toolbar and feedback comments in Non-Canvas LMS’s

[nairiboo]

Problem Statement: Instructors in non-Canvas LMS’s have to use a 1 out of 10 grading scale in the H grading toolbar to translate to the gradebook which is sometimes tough to align if they are using a different grading scale. Also, the ability to add feedback comments is clunky in some LMS’s. Rationale: We want all LMS users to have an optimal grading experience. Impact: Many clients have expressed frustration with the grading toolbar mismatch with gradebook. Proposal: Match the grading toolbar with the gradebook entry. Easier flow to giving private feedback.

[nairiboo]

feedback https://hypothes-is.slack.com/archives/C01B9TRVATZ/p1685546733149069

[nairiboo]

Additional feedback from Support:

Problem Statement: Instructors repeatedly run into the same issues when trying to use the grading bar in the LMS app; we largely accommodate these needs through Canvas SpeedGrader but not for other LMSs. Instructors want to give student private feedback while grading. This is accomplished through grading comments in SpeedGrader. Grading comments do exist in at least some of the other LMSs, and I think there’s an LTI parameter for it. Instructors are often confused by the fact that our grading bar asks them to grade out of 10 points. While it’s true there’s no LTI parameter (that I’ve found) where we can see the actual points​ for an assignment, saying that grading is out of a point total they might not be using is very confusing. Instructors are under the impression that our platform stores data like grades and sometimes want to “re-sync” after they delete grades from their gradebook. This behavior is likely learned from other apps that do store that kind of data and sync it back to the LMS at request.

Rationale: This is important because it provides and confusing and not very professional experience whenever an instructor goes to grade a Hypothesis reading in non-Canvas LMSs. It limits their ability to give feedback, making us less useful than other pedagogical tools, and first time users end up having bad initial grading experiences until they happen to read our KB ​or attend a training.

Impact: Non-Canvas LMSs, even in aggregate, make up a minority of our usage. But I believe we expect D2L usage to grow, and most folks don’t know to differentiate the experience between Canvas and other LMSs when they, for example, share word-of-mouth information with each other on Hypothesis. Proposal:

1. If it’s correct that the LTI standard allows both a grade and a comment to be added to a gradebook we should add commenting functionality to allow instructors to give students better feedback on Hypothesis assignments.
2. We could use each LMSs API to get the actual point total from the LMS, but a sufficient middle group might be to have instructors grade by percentage instead of out of 10 points, which will make sense to them and work the same way in our grading bar.
3. To me this is the least important fix with the biggest lift, but it’s worth considering storing this data if we ever majorly change the LMS product to rely less on LTI and more on storing data in our own ecosystem.

[marcospri]

I've done some testing on LTI1.3 and D2L.

PR at: https://github.com/hypothesis/lms/pull/5479

Based on that all of this seems possible without a huge development cost.

We'd need to test this in other LMSes before committing to anything but I'd argue we should make these LTI1.3 only features and start treating LTI1.1 as a legacy integration.

A rough plan will look like:

Grading comments

• Test the PoC in Blackboard, D2L and Moodle.
• Grading bar UI design to allow, in some cases, displaying a comment field.
• The backend should inform the frontend if the comment feature is available (eventually on for all LTI1.3 installs)
• The backend should send the comment to the grading LMS API when it gets one.
• The backend should send the comment value to the grading bar frontend to get the existing value in our UI.

Total points

• Test the PoC in Blackboard, D2L and Moodle.
• (Ideally) The grading bar configuration could be moved to an async API call, similarly to how we handle the sections/groups API calls.
• The backend should make a query to the LTIA API when showing the grading bar, fetching the maximum score.
• Prominently display the maximum score in the grading bar.
• The backend should send "10" when this feature is not enabled to keep the same behavior as now.

Persist more data around grading.

• Persist scores sent by us to the grading API
• Persist comments sent by us to the grading API
• Persist submissions sent by us to the grading API

For now I'd store this as a bookkeeping exercise, keeping the LMS itself as the "source of truth" for all grading information.

---

Extra details of the quick PoC:

If it’s correct that the LTI standard allows both a grade and a comment to be added to a gradebook we should add commenting functionality to allow instructors to give students better feedback on Hypothesis assignments.

The spec allows for a comment to be sent along side the score.

I've tested this locally in D2L and it gets rendered in the LMS like:

We could use each LMSs API to get the actual point total from the LMS, but a sufficient middle group might be to have instructors grade by percentage instead of out of 10 points, which will make sense to them and work the same way in our grading bar.

We can get the "scoreMaxiumm" from the LTIA API too and based the grading bar on that value when available.

To me this is the least important fix with the biggest lift, but it’s worth considering storing this data if we ever majorly change the LMS product to rely less on LTI and more on storing data in our own ecosystem.

I reckon we should do this in parallel, recording this information is useful for debugging proposes own its own and will be avialble once we need for the product.

[nairiboo]

Thanks Marcos - this looks really promising! Would love to be able to prioritise for next cycle - have added it to the "new" column so it won't get missed during the meeting.

Based on that all of this seems possible without a huge development cost.

Could it be something done in a cycle or less?

[marcospri]

Could it be something done in a cycle or less?

Maybe, more so if all LMSes behave nicely and we don't have to accommodate for a lot of difference between them.

[jeremydean]

I think this is a great project and perhaps an easy win in the balance of priorities.

The only thing I have to add is that the "grading comments" AKA instructor feedback part of this is critical to claiming Hypothesis supports Regular and Substantive Interaction standards.

Part of those regulations focus on "instructor-initiated interactions" and "personalized feedback on assignments" is a key mechanism.

For additional context, see how our competitor Harmonize talks about RSI and feedback. (Link to join group.)

[acelaya]

Adding here the design for the grading comment, from @jaredpdesigns

• By placing buttons in the comment field, we can save the users a click of closing out of the panel and then clicking the "Submit Grade" button
• There are now 3 potential ways of "abandoning" a comment:
  • Click the x icon
  • Click the x Cancel button
  • Click outside the dropdown
• If a user has previously left a comment but wants to update it, primary button in dropdown changes to "Update Comment"

[acelaya]

We had another round of thinking around UX on the comment, and had made decisions:

• "Submit grade" button will be statically "highlighted" in green after successfully submitting (maybe also with a green tick instead of a gray one? I'll sketch something and ask for help if I can't get something nice)
• Since we don't save the grade number and the comment sepparately, but all in one go, the main button inside the comment popover will be labeled as "Submit grade", as it effectively does the same as the other "Submit grade" button.
• We will stop tracking unsaved comments, and instead enforce "submit or discard" via a modal if the instructor tries to change selected student and they didn't submit beforehand.

[acelaya]

I have moved this to "done". Let's keep it open until it's fully released.