Make it possible to grade discussions

e0d commented 1 year ago

At-A-Glance

More information

Product specs

Design files

Kanban board

Stakeholders

Primary Contributors

Arizona State University
OpenCraft

Community release milestones

Quince:
Redwood:

How to contribute

[ ] Has a community backlog with work to pick up?

e0d commented 1 year ago

ASU have provided their use cases/proposed requirements for this feature.

https://docs.google.com/document/d/1qUOELSBMidQ4aJXbTnxibGTfBS_f5Wb9/edit

e0d commented 1 year ago

We discussed some very high-level technical design options in the Data Working Group meeting this morning.

One direction would be to aggregate discussion events in the OARS platform and allow the LMS to access that aggregated data for grades. This option was not favored as it would require pulling OARS in as a requirement.

A counter proposal was to have a plugin in the LMS that listens to forums "events." The plugin could potentially listen to Django signals or the event stream.

Question: should this use the EventBUS?

Tracking log events and signals are generated here.

Signal

comment_created.send(sender=None, user=user, post=comment)

Logged Event

track_comment_created_event(request, course, comment, comment.thread.commentable_id, followed)

Need to know:

Is there a path to listening to events from more than one forum back-end. This would enable graded discussions that was back-end agnostic.
Have changes to the forums experience and choices changed event logs in any way?

I believe that the grading criteria currently specified in the requirements doc can all be covered via signals or the event log.

“Add a Post"
“Reply to message”
How many words each post/reply should contain
The date of the post
The date of the reply

itsjeyd commented 1 year ago

ASU have provided their use cases/proposed requirements for this feature.

https://docs.google.com/document/d/1qUOELSBMidQ4aJXbTnxibGTfBS_f5Wb9/edit

A first proposal for implementing the requirements from this document can be found here:

https://docs.google.com/document/d/1FNlBXYdn9euYDgOq1tNU_XUOdgeWxIXv7Ro0npIpHbs/edit

ProductRyan commented 1 year ago

heads up @ayub02 - can you give this a read and determine if it meets our needs as well?

jmakowski1123 commented 1 year ago

Next steps: -Connect with Aamir and compare the discovery already done with 2U plans

felipemontoya commented 1 year ago

This is an area I'll be happy to collaborate in.

itsjeyd commented 1 year ago

Hey @jmakowski1123 and @e0d, could we get @cassiezamparini (https://github.com/cassiezamparini) added to the openedx organization? She is one of OpenCraft's UX designers and will be taking the lead on this epic starting this week.

I tried to assign her but her name wouldn't show up in the list of suggestions.

CC @ProductRyan @felipemontoya

jmakowski1123 commented 1 year ago

Dropping link to the implementation proposal here so it's all in one place: https://docs.google.com/document/d/1FNlBXYdn9euYDgOq1tNU_XUOdgeWxIXv7Ro0npIpHbs/edit

cassiezamparini commented 1 year ago

Hi @jmakowski1123 hope you're well! I've just followed up with Aamir re the implementation proposal: https://docs.google.com/document/d/1FNlBXYdn9euYDgOq1tNU_XUOdgeWxIXv7Ro0npIpHbs/edit#heading=h.ogslfhqqejyh

Hopefully we get feedback soon so we can discuss next steps :)

cassiezamparini commented 1 year ago

Hey @xitij2000,

Aamir sent the following feedback on the Discovery:

The only thing that stands out at this time is the proposed graded discussion xblock. As mentioned in the proposal, part of BD-38 was meant to replace the discussion xblock with a sidebar and an easier topic creation mechanism (details here). Also, the current discussion xblock uses an older version of the text editor that stores data differently compared to the newer version used in sidebar and discussion MFE, which results in some compatibility issues when editing posts, adding images, urls etc. These compatibility issues are part of the motivation to deprecate the discussion xblocks.

Let me know your thoughts on this? @felipemontoya I know you were interested in collaborating on this too, so I'd love your two cents.

I've asked Aamir if their team is comfortable joining our conversation in Github rather than Slack, so we can all brainstorm together.

cc. @jmakowski1123

xitij2000 commented 1 year ago

The only thing that stands out at this time is the proposed graded discussion xblock. As mentioned in the proposal, part of BD-38 was meant to replace the discussion xblock with a sidebar and an easier topic creation mechanism (details here). Also, the current discussion xblock uses an older version of the text editor that stores data differently compared to the newer version used in sidebar and discussion MFE, which results in some compatibility issues when editing posts, adding images, urls etc. These compatibility issues are part of the motivation to deprecate the discussion xblocks.

I was heavily involved in BD-38 and the effort to move from the Discussion XBlock to the new system that exists today. However, this discovery is not about reinstating the existing Discussion XBlock, or using the existing discussion XBlock. The proposal in the discovery is based on the new discussion configuration mechanism, but involves introducing a new XBlock for graded discussions. The reason for creating a new XBlock is that AFAIK currently the grading machinery needs a XBlock to associate a grade with, so this XBlock can handle. To assign a grade to the discussion, we need an XBlock that will be assigned that grade. The XBlock itself can potentially be something that is auto-injected when enabling/disabling graded discussions. This will then automatically plug into the rest of the grading system so that no changes are needed to the core grading mechanism.

Alternatively, the grading system will need to be updated to either be pluggable, or support grading something other than XBlocks. This is likely to be a much more intrusive and major change.

e0d commented 1 year ago

@xitij2000 Do you have thoughts about how that XBlock would work in practice? What would an author need to do if Discussions should be graded universally in a course? Would the XBlock need to be added to every sub-section for example.

I think it may make sense to do some discovery around opening up the grading system as that would open many additional doors.

xitij2000 commented 1 year ago

@xitij2000 Do you have thoughts about how that XBlock would work in practice? What would an author need to do if Discussions should be graded universally in a course? Would the XBlock need to be added to every sub-section for example.

Here were my thoughts on the above when doing the discovery:

The XBlock can be configured to grade only topics in the unit or subsection it belongs to, or for the whole course.
If you choose to grade universally, then you just add the XBlock in a single place, and it will apply all the grades to that subsection.
Each XBlock can also have a different grading criterion. For example one could grade based on number of posts and responses, while another could grade based on quality. These could both be in different places, and would apply the grade to their respective contexts.

I think it may make sense to do some discovery around opening up the grading system as that would open many additional doors.

Sure, I think discovery is needed here, but I do think having filters here could work. A filter pipeline could be used to have additional plugins inject grades. Each plugin could inject a dict with a unique plugin id, a version hash (so if an external change requires recomputation of grades similar to visible blocks, course version etc), and earned/possible values.

The filter can be called during grade computation and grades produced this way can be persisted in a separate table other than the courseware student module.

If this direction seems more appropriate, then a deeper discovery will definitely be needed.

asadazam93 commented 1 year ago

@xitij2000 I like your idea. However I have one question, how do you plan to handle the case of grading discussions that are related to a specific topic/unit? As I understand the Xblock graded discussion will be added at the unit level itself, how will we connect it to other units/topics?

xitij2000 commented 1 year ago

@xitij2000 I like your idea. However I have one question, how do you plan to handle the case of grading discussions that are related to a specific topic/unit? As I understand the Xblock graded discussion will be added at the unit level itself, how will we connect it to other units/topics?

The XBlock is just a mechanism to allow discussions to participate in grading using existing infrastructure. The mechanism (whether in the XBlock or elsewhere) can expose a UI to select which topics should be considered for grading. For instance, an XBlock in Section1>Subsection1>Unit1 can be configured to consider:

Only the topic from Unit1
Only the topics from Subsection1
All topics across the course Optionally there can be a more extensive UI or more options for selection.

cassiezamparini commented 1 year ago

@asadazam93 Let us know what you think of @xitij2000's thoughts above ^^

If everyone is on the same page, @xitij2000 could you update the Discovery to include anything that wasn't 100% clear (ie. adjust the document to include any queries from this point in our discussion onwards)

ayub02 commented 1 year ago

@xitij2000 thanks for sharing the details. @asadazam93 i'd like to know your thoughts.

I find the user experience (UX) of utilizing an xblock for grading to be potentially challenging for authoring teams. I agree with @e0d regarding the importance of examining the grading system and considering the potential benefits of incorporating grade injection. 2U partners have also been requesting grading for LTI discussion tools. By implementing the ability to inject grades, we can simplify the UX and both edX discussions and LTI discussion tools can make effective use of this feature.

jristau1984 commented 1 year ago

Just wondering if the proposed implementation can get a no-code Figma prototype built out so that users could experience what is being proposed? This would enable real user feedback collection and be able to see if the UX proposed really meets user needs sufficiently before we invest too much in either direction...

cassiezamparini commented 1 year ago

I'd 100% agree with that, if that's an option. It will also allow us to iterate far faster on various directions.

cassiezamparini commented 1 year ago

@xitij2000 What do you think about prototyping wireframes of the proposed solutions first? This might be a good starting point before a technical discovery is output? We'll just need to put together a UX discovery together first.

felipemontoya commented 1 year ago

thanks @cassiezamparini for tagging me in this. Only now I could get up to speed in here.

From the use cases that we need to support, the proposal with the Xblock would cover it. Adding a filter to the grading infrastructure however could also open the door for including completion API in the grading as well. cc @mariajgrimaldi this is something we could start a discovery in.

xitij2000 commented 1 year ago

2U partners have also been requesting grading for LTI discussion tools.

I think LTI already supports grading if I remember correctly. So should already be a way to grade LTI-based discussions. However this will only work for discussions embedded in an LTI block in the course, not for course-wide discussions.

@xitij2000 What do you think about prototyping wireframes of the proposed solutions first? This might be a good starting point before a technical discovery is output? We'll just need to put together a UX discovery together first.

If that would be useful, sure. However, I think in this case the UX is kind of independent from the implementation. i.e. whether if it is the XBlock or the plugin approach, they can both handle roughly the same UX. Adding an XBlock for grading can just be an internal process.

mariajgrimaldi commented 1 year ago

@xitij2000's proposal of using filters to extend the grading capabilities is pretty interesting. So much, we made a little discovery about it here. Basically, we propose adding a filter for checking additional conditions besides the course grade to determine whether the student passed the course. By checking this, we could add conditions like: number of posts published -- by using discussions utilities-- or % of course visited -- by using the completion API --.

Feel free to comment on the document or here about this POC.

cassiezamparini commented 1 year ago

@jmakowski1123 and @ayub02 Just checking in to see if you've made / or are planning to make any headway on the product spec for the MVP?

jmakowski1123 commented 1 year ago

Thanks for checking in @cassiezamparini , it's on my to-do list for July!

cassiezamparini commented 1 year ago

Hey @ayub02 and @jmakowski1123, just checking in on the progress of the product spec? :)

jmakowski1123 commented 1 year ago

Update 8/16 - Still on @jmakowski1123 to kick off spec work!

jmakowski1123 commented 6 months ago

Dupe ticket, closing in favor of https://github.com/openedx/platform-roadmap/issues/351

openedx / cs_comments_service