An automated assignment grading system that leverages LLMs and AI to enhance grading efficiency and reliability. It includes modules for data input, criteria definition, AI integration, consistency checks, and comprehensive reporting, aimed at improving educational outcomes.
3
stars
3
forks
source link
Automated Assignment Grading System with AI Model Evaluation
Objective:
Develop an automated assignment grading system to reduce manual grading fatigue and improve efficiency.
Evaluate the reliability of various AI models in grading assignments as part of a research initiative.
Project Steps:
Project Setup:
Establish a project timeline and milestones.
Identify the necessary technological stack and resources, including RAG, LangChain, and LLM models.
Data Input Module:
Develop an input module that accepts a zip file containing text-based assignments (Word, PDF, Python notebooks).
Design a mechanism to extract and process text from these formats for grading.
Grading Criteria Definition:
Implement a system allowing graders to define specific grading criteria and processes for each assignment type.
Ensure the system supports diverse grading metrics and workflows tailored to different assignments.
AI Grading System:
Integrate AI automation flows to grade assignments based on the predefined criteria.
Employ multiple AI models and compare their grading outputs to ensure reliability and consistency.
Grading Consistency and Verification:
Define a protocol for grading each assignment multiple times to verify consistency.
Develop an algorithm to compare grading outcomes across iterations, flagging significant discrepancies for review.
Flagging System:
Create a web interface to display flagged assignments, highlighting variations in grading.
Allow graders to review and adjust grades for flagged assignments, ensuring accuracy.
Result Compilation and Reporting:
Generate comprehensive reports detailing each assignment's grade, confidence levels, and variation across grading iterations.
Include constructive feedback based on grading metrics, offering insights into areas of improvement for students.
Testing and Iteration:
Conduct thorough testing of the system with a variety of assignment samples.
Gather feedback from potential users and refine the system based on this input.
Future Expansion:
Plan for the incorporation of more complex assignment types, including those with embedded images or code, in subsequent phases of the project.
Documentation and Dissemination:
Document the development process, system architecture, and user instructions.
Prepare a paper or presentation to share the research findings on AI model reliability in grading.
Expected Outcomes:
An operational automated grading system that enhances grading efficiency and consistency.
Insights into the effectiveness and reliability of different AI models in the educational grading context.
Detailed Documentation