search

openedx / platform-roadmap

Tracking the maintenance, enhancement, and advancement of the Open edX project.
11 stars 1 forks source link

Integration for Turnitin #253

Open jmakowski1123 opened 1 year ago

jmakowski1123 commented 1 year ago

At-A-Glance

Supported integration point for Turnitin, a third-party plagiarism tool.

More information

Product specs

Design files

Kanban board

Stakeholders

Primary Contributors

Community release milestones

How to contribute

github-actions[bot] commented 1 year ago

Thanks for your submission, @openedx/open-edx-project-managers will review shortly.

jmakowski1123 commented 1 year ago

Based on Spanish proposal needs

felipemontoya commented 1 year ago

1. Is there an existing issue for this?

2. What new feature or functionality would you like to request?

Create an OpenAssesment step that is extendable via openedx-filters. This in order to connect the ORA open responses with academic integrity tools such as turnitin.
[optional] create an openedx-event after grading to allow developers to extend ORA even further.

3. What product area does this feature affect?

MS, spedifically the ORA2 type questions.

4. Please describe the scope of the feature.

The proposal is to create an openedx-filter that receives the text of the ora response and other context information and passes it to a configurable pipeline of possible implementations.

The response from the filter will contain the grade for the step.

In a separate plugin we would also implement a hook pipeline to send the learner's response to turnitin. This implementation would be completely modular and serve as an example for other to write hooks to connect to other tools and even to grade responses by using LLMs.

5. Please describe why you see a need for this feature.

Open ended questions are very flexible and very useful for evaluation, but many students have used them with cheated responses.
With the availability of LLM tools like chatGPT this could get sensibly worse.

6. Please describe the potential impact and/or value of this feature

- Knowing that answers will be passed to anti plagiarism tools would nudge the learners away from cheating.
- Automatic reviews could be used to give immediate feedback.
- Opens a stable extension point for more innovation in the ORA space.

7. Please provide 2-3 use cases and/or user stories in support of this feature

- Plagiarism and cheating review before the staff review in ORA. (requires a plugin for each third party tool)
- Services like ChatGPT could be use to grade or review the open ended response (this would require a plugin that is not covered in the Spanish Universities requirement)

8. Any additional information you'd like to provide?

This request comes from the Spanish Universities consortium.
Specifically the first requirement from the "Partida 4" section of the tender.
ProductRyan commented 1 year ago

heads up @Daniel-hershel

nandodev-net commented 1 year ago

Hello, my name is Fernando Gonzalez, I work with Edunext, here's an integration proposal

Turnitin <-> ORA integration proposal workflow

Prior to describing the workflow as conceived to date, there are some points that should be highlighted:

Turnitin_ Integration Diagram

As shown in the image, the flow for carrying out an individual evaluation from Turnitin towards an ORA deliverable would be conceived through the following stages:

Stage 1: EULA

EULA refers to terms and conditions that will be displayed to both the instructor and the student, and whose acceptance is imperative when using Turnitin's services. To obtain this, Turnitin provides us with an endpoint that returns a URL, which we can link to a notification for users who need to accept these terms and conditions in order to proceed whit the submission.

Interaction with the EULA on the part of the instructor:

Interaction with the EULA on the part of the student:

Stage 2: Submission

In this stage, the student will carry out the upload of their file in a conventional manner, that is:

At the moment the student presses the Submit button,

It is expected that a subsequent step will appear with the link to the EULA and the respective acceptance text.

Once accepted, proceed to prepare the submission flow to Turnitin using the endpoints provided in the docs. (Suggestion: in the case where more than one file is uploaded, a loop of the following steps should be created, along with a file extension filter).

  1. Create a Submission through the endpoint: Create Submission

    POST https://$TII_API_URL/api/v1/submissions

    Note: remember to include the EULA block in the JSON.

  2. Attach the file to the deliverable through the id returned by the previous step, using the endpoint: Upload Submission File Contents

    PUT https://$TII_API_URL/api/v1/submissions/{id}/original

    1. Wait for Turnitin's webhook to notify us that the deliverable has been processed and is ready for the next step.

      Note: it was thought that from this step, and until obtaining the Turnitin grade, the student may see a "processing evaluation" display (this should be evaluated as an idea).

  3. Once the webhook arrives, the system must automatically request Turnitin to start processing the report, this would be through the following endpoint: Generate Similarity Report

    PUT https://$TII_API_URL/api/v1/submissions/{id}/similaritys

    1. Again, we will be waiting for a webhook to notify us when the report is ready. Once this happens, we will have at our disposal a series of actions (generate a PDF report, request the percentage of matches, generate a report viewer), for now, the one to be taken is to obtain the match percentage through the endpoint: Obtain Information from the Similarity Report

      GET https://$TII_API_URL/api/v1/submissions/{id}/similarity

With this, the workflow for a student to submit a file to Turnitin and receive a similarity grade is defined, which is what is currently being sought.

juancamilom commented 1 year ago

Here is a quick recap / update on this issue to make sure that the conversation stays alive. The edunext / unidigital team is very mottivated and committed to make this contribution and We are already collaborating with Turnitin officials and technical teams to overcome the tecnical challenges for an integration.

We are ultimately interested in enabling an integration beetween open edX (ORA-2 in particular) and turnitin that would allow these things to happen:

A) That open edx holds the turnitin Core API credentials either as a setting inside the ORA component or as a setting in the course.

B) that the files submitted by a learner to the ORA problem can be made available to be sent to turnitin

C) that said file(s) are sent to turnitin for review and any immediate response from the turintin API is handled appropriately. for example if the credentials are not correct, there is a timeout, the file is rejected, etc.

D) that when the request is accepted by turnitin, and while turnitin has not provided a similarity reports, an indication is shown to learners and instructors in the open edX side

E) that when turnitin provides a response, the link to the viewer of the corresponding similarity report can be seen in the open edX UI for learners and instructors.

We are initially focusing the effort in making sure all the interactions facing turintin work as required, and building this code into an open edx plugin, with the idea that it could potentially be leveraged in places other than ORA (SGA, or a different xblock for example).

However, in order to reduce the uncertainty on the ORA´s side of things, we´d like to ask and validate the following:

1) what would it take for ORA to make the file(s) submitted by a learner available for the turnitin use case (ideally via an open edX event)? 2) can we propose an extension point (an open edX filter) to be added to ORA that would allow us to display the response we get from turnitin to the learner and the instructor? 3) does is make sense to build this extension point into the legacy experience, or to wait a bit and aim to contribute this as part of a Re-imagined Learner UX for ORA ?

In case (3) lands in the legacy interface, we imagine something like this for the learner: image

and something like this for the instructor: image

@e0d, @Daniel-hershel and @mattcarter your feedback and thoughts on questions 1, 2 and 3 is highly appreciated as always.

e0d commented 1 year ago

Hi @juancamilom I think that the original ORA design entertained extension via customization of and addition to the workflow concept that is used for grading submission. So, while I love events and filters, it may be easier and more natural to add a workflow step that supports integration with TurnItIn. There are a couple examples in branches that may be useful:

https://github.com/openedx/edx-ora2/blob/efischer/credo_replay/openassessment/assessment/api/ai_worker.py https://github.com/openedx/edx-ora2/blob/will/ai-cache-classifiers/openassessment/assessment/api/ai_worker.py

I would assume this style of machine assessment integration would have natural methods for access the complete submission, including uploaded files, but this is an assumption at this point.

Adding that leveraging the event bus rather than celery would probably be preferable.

juancamilom commented 1 year ago

Thank you very much @e0d .
We initially thought that the natural way of integrating turnitin was via an additional assessment step, but we started looking in other directions once we learned that the response turnitin provides is not a quantitative assessment of the submission, but instead a qualitative report that diagnoses the writing in the file against internet sources. It is true that this qualitative result also has a similarity score, but it may not be the case that this criteria will fit into the ORAs rubric. It is also true that you can create rubrics in turnitin and have the teachers provide a staff assessment score in turnitin instead of providing it in ORA, but it seems to be an unnecessary burden.
The use case the spanish consortium is after is more directed towards having the similarity report be a tool to help learners reflect on their own work through the lens of similarity and to give instructors additional input they may or may not use when deciding the staff assessment based on the problems rubric.

However, now that you bring our attention back to the assessment steps, I'm wondering whether all assessment steps need to be linked to an score. The training step seems to be one that is not really linked to a score, but to a true or false scenario. We can perhaps explore this route in order to model the turnitin step as an unskippable step, without scoring and where success means that a similarity report was indeed received from turnitin. I´ll make sure the team explores this path and keep you posted.

e0d commented 1 year ago

@juancamilom yes, what you say about the training step seems correct to me. I does feel like the other work that's being reviewed to adjust the weighting of grading steps could provide the ability to have a grading step with 0 weight for the final grade as well.

juancamilom commented 1 year ago

Quick update about this ongoing work. We have finished the implementation of all the pieces facing towards turinitin.

This provisionally lives inside an independent Xblock and can be explored here: https://github.com/eduNEXT/platform-plugin-turnitin/pull/4

Our next goal is to dive deep into the challenges of adding an additional ORA step, that is:

I've published the project in confluence as well following the product working group guidelines, here: https://openedx.atlassian.net/wiki/spaces/COMM/pages/3891855361/Assessment+step+with+external+tools+-+Turnitin

as always, your feedback is much appreciates @e0d , @Daniel-hershel , @mattcarter , @jmakowski1123