search

philchalmers / mirtCAT

Computerized Adaptive Testing with Multidimensional Item Response Theory
http://philchalmers.github.io/mirtCAT/
89 stars 28 forks source link

Strange results in simulation #48

Closed janstochl closed 11 months ago

janstochl commented 11 months ago

Dear Phil, first, thanks for your amazing work on mirt and mirtCAT packages, I could not live without them.

I have a question on strange results that I see when using EAP in MCAT simulation. In a nutshell, I see that the administration of items is finished after response to single items. This only happens when EAP is used (MAP and ML are OK. the reason why the test is finished that the SE_theta drops to ridiculously small values (for all dimensions) after response to single item. Below is the descriptive stats on number of administered items from administrating the test to 1000 simulated individuals .

image

You see that this happens basically for every EAP administration.

When I look into the single simulee theta_se_history I see that the SEs dropped to very low numbers after answering only single question, such as in the example below.

image

Do you have any clue why I see what I see? Your kind insights would be much appreciated, thanks for your response in advance.

Kind Regards, Jan

philchalmers commented 11 months ago

Hi Jan,

Thanks for the report. I can't think of any good reason why the EAP SEs would suddenly drop to 0, so this must be a bug somewhere in the code. Could you provide a brief reprex of the issue?

janstochl commented 11 months ago

Thanks for response Phil, I will try to prepare it later today and send it over to you. Thanks. Jan

Dne 12.10.2023 14:34:46, Phil Chalmers @.***> napsal:

Hi Jan,

Thanks for the report. I can't think of any good reason why the EAP SEs would suddenly drop to 0, so this must be a bug somewhere in the code. Could you provide a brief reprex of the issue?

— Reply to this email directly, view it on GitHubhttps://github.com/philchalmers/mirtCAT/issues/48#issuecomment-1759626377, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APXTWHHH5TTHLB3ODMHHUFDX67WXHAVCNFSM6AAAAAA54HLOD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJZGYZDMMZXG4. You are receiving this because you authored the thread.Message ID: @.***>

janstochl commented 11 months ago

Dear Phil, please find reproducible example attached. I am sending item parameters in .RData, the actual script is in separate file. The simulation takes about 10 mins but you may reduce the number of adaptive criteria if you want to reduce the time (I wanted to demonstrate that the issue is the same regardles of criteria used).

Please let me know what you think. And of course, huge thanks for your time you spend on this.

Kind Regards, Jan

Dne 12.10.2023 14:34:46, Phil Chalmers @.***> napsal:

Hi Jan,

Thanks for the report. I can't think of any good reason why the EAP SEs would suddenly drop to 0, so this must be a bug somewhere in the code. Could you provide a brief reprex of the issue?

— Reply to this email directly, view it on GitHubhttps://github.com/philchalmers/mirtCAT/issues/48#issuecomment-1759626377, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APXTWHHH5TTHLB3ODMHHUFDX67WXHAVCNFSM6AAAAAA54HLOD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJZGYZDMMZXG4. You are receiving this because you authored the thread.Message ID: @.***>

philchalmers commented 11 months ago

Hi Jan,

I don't see any attachments in the email or Github locations. Can you try uploading again?

janstochl commented 11 months ago

I sent via email, but it looks github is not supporting R script and RData files as attachments. I tried here and also cannot upload them. Is there any other way how I could get the files to you please? Sorry I am not using github that frequently.

philchalmers commented 11 months ago

No problem. Can you send the files directly to my gmail address? @.***

Phil

On Fri, Oct 13, 2023 at 9:02 AM janstochl @.***> wrote:

I sent via email, but it looks github is not supporting R script and RData files as attachments. I tried here and also cannot upload them. Is there any other way how I could get the files to you please? Sorry I am not using github that frequently.

— Reply to this email directly, view it on GitHub https://github.com/philchalmers/mirtCAT/issues/48#issuecomment-1761485762, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJXEYBCHZCRASTJER4WJPDX7E3XFAVCNFSM6AAAAAA54HLOD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRRGQ4DKNZWGI . You are receiving this because you commented.Message ID: @.***>

janstochl commented 11 months ago

Phil, github is masking your email so I cannot see your email address, I basically see *@****

janstochl commented 11 months ago

I think I found your gmail at York university website. Sending the files there. You will receive it from my cambridge uni address (which starts strangely with js883). Please let me know whether thi files have arrived. Thanks.

philchalmers commented 11 months ago

Wow, that's needlessly aggressive of Github.... Files received, thanks for keeping at it!

janstochl commented 11 months ago

perfect, thanks!

philchalmers commented 11 months ago

Hi Jan,

I see the problem here, though it's not a bug. The issue is that you have a rather high dimensional model so numerical integration is going to be a major issue. In general, I don't recommend using EAP estimators past 4 dimensions given that the integration precision must drop to accommodate the exponential growth of the grid (e.g., 6 factors and even 11 quadrature that's 11^6 = 1.7 million points to evaluate!). The package currently uses mirt's fscores() defaults, which admittedly are a little low (for 6 factors only 3 quadrature were used; I increased this value to 5 for the next version), however my main recommendation is to either use MAP estimates or try your luck with quasi-Monte Carlo integration to get a sparce integration grid to work with.

At this point, I think for safety reasons I should probably throw a message to the user so that they are immediately aware of the EAP problems for high-dimensional models.

philchalmers commented 11 months ago

Latest dev version of mirt seems to fix this issue, however a warning is now printed in mirtCAT to deter the use of EAP estimators for high-dimensional models. Thanks for bringing this to my attention.

janstochl commented 11 months ago

Thanks a ton Phil for demystifying this so quickly. Good to know about limitations of EAP. Makes sense and thanks for implementing the warning.

Best wishes, Jan

Dne 13. října 2023 16:09:28 Phil Chalmers @.***> napsal:

Latest dev version of mirt seems to fix this issue, however a warning is now printed in mirtCAT to deter the use of EAP estimators for high-dimensional models. Thanks for bringing this to my attention.

— Reply to this email directly, view it on GitHubhttps://github.com/philchalmers/mirtCAT/issues/48#issuecomment-1761677886, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APXTWHC7VAGKNEHXSCV5GSTX7FKSHAVCNFSM6AAAAAA54HLOD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRRGY3TOOBYGY. You are receiving this because you authored the thread.Message ID: @.***>