search

philchalmers / mirt

Multidimensional item response theory
https://philchalmers.github.io/mirt/
199 stars 75 forks source link

MIRT survey-weights option breaks M2 calculations #174

Closed dcoven closed 4 years ago

dcoven commented 4 years ago

The M2() limited information statistics do not properly account for survey weights in the model. The following calculates a model from an expanded table and from using weights, yet M2 is wildly different for the two methods.

LSAT6_Expanded <- expand.table(LSAT6) dim(LSAT6_Expanded) mod_Expanded <- mirt(LSAT6_Expanded, 1, "Rasch") M2(mod_Expanded )

mod_Survey<- mirt(LSAT6[,1:5], 1, "Rasch", survey.weights =LSAT6[,6] ) sum(LSAT6[,6]) M2(mod_Survey)

philchalmers commented 4 years ago

Can you describe why this behaviour is unexpected? Survey weights are used to evaluate more optimal parameter estimates given population demographic information via weights. Fit statistics like M2 are evaluating whether the current data were plausibly obtained according to the structure implied by the model. If the model parameters were happen to be chosen by a population with very different characteristics (i.e., the weights have a large impact) then the current data should seem to fit less in accordance with this structure.

dcoven commented 4 years ago

Good point. The reason I didn’t expect this is that I’m using sampling weights to work with expected cell probabilities, rather than true population samples.

Do you know if there is another way to tweak MIRT to work with probabilities for response patterns, rather than sample data?

Thanks for your time!

Daniel Coven

From: Phil Chalmers notifications@github.com Sent: Sunday, February 16, 2020 10:49 AM To: philchalmers/mirt mirt@noreply.github.com Cc: dcoven github-real@messagewell.com; Author author@noreply.github.com Subject: Re: [philchalmers/mirt] MIRT survey-weights option breaks M2 calculations (#174)

Can you describe why this behaviour is unexpected? Survey weights are used to evaluate more optimal parameter estimates given population demographic information via weights. Fit statistics like M2 are evaluating whether the current data were plausibly obtained according to the structure implied by the model. If the model parameters were happen to be chosen by a population with very different characteristics (i.e., the weights have a large impact) then the current data should seem to fit less in accordance with this structure.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/philchalmers/mirt/issues/174?email_source=notifications&email_token=AOSIUTJ2USKZEACXDI55KM3RDF4B5A5CNFSM4KWDUPMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4NPAA#issuecomment-586733440 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AOSIUTLOIPT7EMHISJTRHG3RDF4B5ANCNFSM4KWDUPMA . https://github.com/notifications/beacon/AOSIUTLIBON7FVYD4MHPAWDRDF4B5A5CNFSM4KWDUPMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4NPAA.gif

philchalmers commented 4 years ago

What you did was correct; the weights correspond to unique response patterns by weighting by some expected probability according to the pattern in the population. So I'm not quite sure what you're looking for here. In any event, I don't see this as an issue at this point, so I'll close this for now.