Optimal-Learning-Lab / LKT

12 stars 2 forks source link

How to interpret the pred values inside the "newdata" of the model? #17

Open erkaner opened 11 months ago

erkaner commented 11 months ago

I am having hard time to correctly interpret the values in the pred column of the "newdata", a component not explicitly outlined in the available Learning Knowledge Tracing (LKT) resources.

Would it be appropriate to deduce that these values represent predictions regarding whether a student would answer a question correctly (predicted probability ≥ 0.5) or incorrectly (predicted probability < 0.5)? That means, for each question and for each student these predictions are computed, right? Are these predictions supposed to get better when moving forward to next items (as more data gets accumulated during the training)? Additionally, my understanding is that the R2 square value (the model fit) depends on the accuracy of these predictions. Is this interpretation accurate?

Moreover, I'm curious about the interplay between students' knowledge levels (or ability) and these predictions. Any insights you could provide would be greatly appreciated.

imrryr commented 11 months ago

It is the prediction from the logistic regression. The prediction, in some sense, is the model. It represents what the model might have expected would happen at any observation, given its parameters. Some of the features result in the model tracing knowledge, so it tends to improve as practice accumulates. The R2 value is computed directly from the accuracy of the predictions. It is computed using McFadden's method.

You might expect that the predictions would be higher if a student performed better, and the model had a term that allowed it to represent the student's ability in some way.

erkaner commented 11 months ago

Thank you for providing a clear explanation. Given this understanding of the 'pred' column in the 'newdata' component of Learning Knowledge Tracing (LKT) resources, I'm interested in clarifying a specific scenario. Could it be assumed that the prediction corresponding to the final question or item associated with a particular skill for a given student represents their knowledge level of that skill? Or, should I take the average or weighted average of these predictions to obtain an estimate of the student's proficiency in that skill. My efforts to extract individual skill-level information from the model's outputs have led me to this conclusion. I would greatly appreciate your insights on this interpretation.

imrryr commented 11 months ago

Well, yes, some weighted average across items may be needed. If you take the last one, it is biased by whether that item is easy or hard. I'm not sure if there is any universal way to do this. If I were to do it, I might recompute the probabilities for the last few observations using the average difficulty of the item, then I would weight the observations by how recent they were... But that's a bit ad hoc.