Request for Multidimensional IRT

Djokovic0311 commented 3 years ago

Hey @jplalor @EntilZha!

We are working on multidimensional IRT. Could you please add commits for related to multidimensional IRT if convenient? It will be extremely helpful for our next-step heuristics ideas! Thanks in advance.

EntilZha commented 3 years ago

Hey, thanks for the interest! The model is actually already in the repo here https://github.com/nd-ball/py-irt/blob/master/py_irt/models/multidim_2pl.py, but it might not be quite as well integrated (its tricky since the 1d models have 1d parameters and multidim has multidim, so the shapes don't match). You might also look at the paper repository where that model is from here https://github.com/facebookresearch/irt-leaderboard

Djokovic0311 commented 2 years ago

Thanks for your kind reply! I will look at those resources.

jplalor commented 2 years ago

Let us know if you have any questions!

Djokovic0311 commented 2 years ago

Hi! @EntilZha! Could I ask what 'loc' and 'scale' mean in the multi-2PL model? I saw they two in the codes and found that they together form the multidimensional parameter. Thanks a lot!

EntilZha commented 2 years ago

@Djokovic0311 in general, loc refers to the center of the distribution which is the value you generally want and scale refers to how wide it is/variance/etc.

Djokovic0311 commented 2 years ago

@EntilZha Thanks a lot for your explanation! And can we create clusters from MIRT analysis (not post-hoc clustering)? We are trying to find some interesting examples according to the MIRT and IRT values.

EntilZha commented 2 years ago

@Djokovic0311, I'd say yes and no. So the parameters are multidimensional so one could cluster them, but in our research paper (https://aclanthology.org/2021.acl-long.346/) we experimented with this in the Appendix (Figure 15) with limited success. There are one or more things which could be causing this issue:

There isn't anything in the model that encourages parameters values to concentrate in a particular dimension, or not. You could think of this as despite the fact that we would want "mathiness" to correspond to a dimension, "mathiness" might be smeared across multiple dimensions.
Another issue pointed out by https://twitter.com/kchonyc/status/1445406119803895820 is that the skill and difficulty parameters in the multidim case don't need to necessarily be tied, so also the dimensions for say "mathiness" might not be the same across skill and difficulty.

We're working on some ideas that might fix this problem and make multidim more amenable to this type of analysis, but don't have anything yet. Open to chatting more on this though!

Djokovic0311 commented 2 years ago

Okay! I got it. We will look at that idea as well. Thanks for your detailed explanations!

nd-ball / py-irt

Request for Multidimensional IRT #26