Analyze and Evaluate Rubrics Generated by the Automated Rubric Generation Chatbot

[parthasarathydNU]

Objective

The purpose of this issue is to conduct a thorough analysis of the rubrics generated by our automated rubric generation chatbot. We need to evaluate whether these rubrics adhere to established educational and technical guidelines that make them effective for both manual and automated grading systems. This analysis will help ensure that the rubrics are clear, detailed, and consistent, thereby enhancing their utility in educational settings.

Resources to successfully complete this task:

• Refer the current version of the system prompt for reference.
• Read through the Rubric generation philosophy to learn more about what makes a good rubric.
• Refer Prompt Engineering Guide to get more knowledge on how to build a robust chat application.
• Refer OpenAI's strategies for prompt engineering

Tasks

1. Collect Rubric Samples: Gather a diverse set of rubrics generated by the chatbot across various types of assignments and disciplines.

2. Define Evaluation Criteria:

   • Clarity: Rubrics should be easily understandable, with no ambiguous language.
   • Detail: Each rubric criterion should provide specific, measurable indicators of performance.
   • Consistency: Similar performance levels across different rubrics should carry similar descriptions and point allocations.
   • Alignment with Educational Standards: Rubrics should align with accepted educational practices and objectives.
   • Suitability for Automation: Criteria should be defined in a way that allows for automated evaluation (e.g., clear thresholds, quantifiable outcomes).

3. Review Process:

   • Manual Review: Conduct manual reviews of the collected rubric samples to assess adherence to the above criteria.
   • Feedback from Educators: Solicit feedback from a panel of educators on the clarity, usability, and educational value of the rubrics.
   • Automated Analysis: Use text analysis tools to assess the consistency and specificity of language used in the rubrics.

4. Document Findings:

   • Compile the findings into a report detailing how well the chatbot-generated rubrics meet the evaluation criteria.
   • Include examples of both well-constructed and problematic rubrics to illustrate common issues and strengths.

5. Recommend Improvements:

   • Based on the analysis, recommend specific changes to the chatbot’s rubric generation algorithms to improve clarity, detail, consistency, and automation suitability.

Expected Outcomes

• A detailed report on the quality and effectiveness of rubrics generated by the chatbot.
• Actionable recommendations for enhancing the rubric generation process.
• Improved satisfaction among educators and students using the rubrics for grading and feedback.

Impact

This analysis is crucial for ensuring that our automated rubric generation chatbot produces high-quality educational tools that aid in fair and effective assessment practices. By ensuring our rubrics meet high standards, we can significantly enhance the learning and teaching experience.

[parthasarathydNU]

First version of chatbot

https://udify.app/chat/9NkhrQPMEDTqGxYA

Feedback :

• [ ] Didn't ask about Weightage distribution
• [ ] Have feature to upload assignment to be graded (provide some sample assignments for people to test the app )
• [x] Add Conversation starters
• [x] The open conversation from the bot is unrelated to assignment grading
• [ ] Update documentation and redirect users to the GitHub repo from the app / make the app informative
• [ ] Document learning from programmatic prompting
• [x] add link for users to navigate to this issue and add a comment
• [ ] Conversation stops in between and I have to type in continue for the bot to continue generating its content
• [ ] Provide better sample assignments to grade

[parthasarathydNU]

Chatbot Feedback Template

User Information (Optional):

• Name:
• Affiliation:
• Role (e.g., Educator, Student, Developer):

Feedback Submission Date: [Insert Date]

Link to Chatbot Session: [Insert URL]

1. Overall Experience:

• How would you rate your overall experience with the chatbot? (Poor, Fair, Good, Excellent)
• What did you like most about your interaction with the chatbot?

2. What Went Well:

• Please describe aspects of the chatbot that met or exceeded your expectations.
• Were there specific features or interactions that were particularly helpful or enjoyable?

3. Areas for Improvement:

• Were there aspects of the chatbot that could be improved? Please specify.
• Did you encounter any issues or frustrations during your interaction? If so, please detail them.

4. Feature Suggestions:

• Are there any features or functions you believe should be added to enhance the chatbot experience?
• How could the chatbot be more helpful or easier to use?

5. Additional Comments:

• Any other feedback, comments, or suggestions not covered above?

6. Consent to Use Feedback:

• Do you consent to us using your feedback to improve the chatbot and related documentation? (Yes/No)

[parthasarathydNU]

Sample Chat GPT conversation for next iteration of the prompt

https://chat.openai.com/share/eb246711-4749-4d8e-98a6-138b5d8b0e86