Closed bassner closed 2 weeks ago
Within the HybridD3 project, we're investigating the influence of presenting statistics and metrics directly to learners versus having an AI LLM dialogue about their performance. Initial steps for tracking user events like exercise openings and lecture material accesses have been established in #7680.
The "science" subsystem now tracks new user events, but we're not yet using this data to generate advanced statistics. Furthermore, the current metrics do not fully meet the project partners' requirements from the education department for our experiment.
Time Difference Between Last Submission and Exercise Due Date: Measure how late students complete assignments relative to the due date.
Time Difference Between Exercise Start and Last Submission: Assess the duration students actually spend on exercises from start to completion.
Time Difference Between Release Date and Exercise Start: Assess the time students wait to start an exercise.
For the above metrics, provide averages for exercises grouped by the same start and due date day, in both absolute values (seconds) and relative values (% of time between release and due date).
Number of Submissions per Exercise: Track the total of submissions, grouped by exercise ID.
Number of Exercises Submitted weekly: Count how many different exercises a student submits per week, both as an absolute number and relative to the total number of active exercises that week. active = start date <= current date <= due date
.
For each of the metrics above, make a differentiation for a) difficulty level and b) type of the exercise. Also, make a filter for final submissions that have <50% answered or <30% correct if it is auto-graded.
(if possible) Track content Access after initial open Track when and how long a tab with the content (the pdf or video or task) was actively viewed.
To support both direct student feedback and integration with AI-mediated dialogue systems, the metrics should be accessible through:
A REST API endpoint for frontend requests, enabling data visualization (detailed UI/UX design will be addressed separately).
Service method calls for internal use, particularly for integrating with the course-level IRIS chat pipeline to enrich AI context.
This approach ensures that the data can efficiently serve both direct student engagement and our future course level IRIS chat pipeline.
Context
Within the HybridD3 project, we're investigating the influence of presenting statistics and metrics directly to learners versus having an AI LLM dialogue about their performance. Initial steps for tracking user events like exercise openings and lecture material accesses have been established in #7680 .
Problem
The "science" subsystem now tracks new user events, but we're not yet using this data to generate advanced statistics. Furthermore, the current metrics do not fully meet the project partners' requirements from the education department for our experiment.
Desired User Metrics
Exercise Metrics
Time Difference Between Last Submission and Exercise Due Date: Measure how late students complete assignments relative to the due date.
Time Difference Between Exercise Start and Last Submission: Assess the duration students actually spend on exercises from start to completion.
For the above metrics, provide averages for exercises grouped by the same due date day, in both absolute values (seconds) and relative values (% of time between release and due date).
Weekly Activity Metrics
Number of Submissions per Exercise: Track the weekly total of submissions, grouped by exercise ID.
Number of Exercises Submitted to: Count how many different exercises a student submits per week, both as an absolute number and relative to the total number of active exercises that week.
active = start date <= current date <= due date
.Lecture Unit Metrics
Data Access
To support both direct student feedback and integration with AI-mediated dialogue systems, the metrics should be accessible through:
This approach ensures that the data can efficiently serve both direct student engagement and our future course level IRIS chat pipeline.