search

DIDSR / iMRMC

iMRMC: Software to do multi-reader multi-case analysis of reader studies
http://didsr.github.io/iMRMC/
Other
22 stars 17 forks source link

Power estimation using Paired split plot design #184

Closed PSIOT closed 2 hours ago

PSIOT commented 2 weeks ago

Hello, we have done a pilot study to be able to validate the sample size of a future MRMC with paired split plot design. Using the results of the pilot to estimate the power a warning message appears on iMRMC; “‘DF_BDG is below a minimum…’. Can we accept the calculated power or is it better to choose the 'Size MLE' option for power calculation? Thank you image

brandon-gallas commented 2 weeks ago

I would be happy to answer your question. Please update your GitHub profile with your name and affiliation. Thanks.

PSIOT commented 2 weeks ago

My profile has been updated. Thank You

brandon-gallas commented 1 week ago

It looks like you are using the java gui. This is a static piece of software that is no longer being maintained. I recommend that you use the R package moving forward. You can find information here: iMRMC: Software to do Multi-reader Multi-case Statistical Analysis of Reader Studies | Center for Devices and Radiological Health (fda.gov).

To be clear, you are not giving me much information.

The DF_BDG are estimated to be ~12. This is healthy. The warning about the degrees of freedom (DF) is not a problem. If the DF estimates fall below the lower bound, they are set to the lower bound. The DF estimates have uncertainty in them, especially in cases where data is limited.

If DF_BDG were below 3, this estimate is saying your data is equivalent to 3 observations warning would be a red flag that your data was not healthy. This often comes when you have few readers or the readers are highly correlated.

I would avoid using the MLE results. They are not validated when the study design is not fully crossed, and I’ve observed weird results in such cases. Your question and others are nudging me to remove the MLE results completely from the current software.

PSIOT commented 1 week ago

Thank for your answers. I did not find any package on R for power calculation (or sample size) in case of paired split-plot design. Did i miss something?

brandon-gallas commented 1 week ago

We do not have that functions ready to go. You have to put your statistician hat on and use the components of variance that the software provides. Here is an example of sample size estimation.

I would also recommend that you review this paper to understand the components of variance under a paired split-plot study design. W. Chen, Q. Gong, and B. D. Gallas, “Paired split-plot designs of multireader multicase studies,” Journal of Medical Imaging, vol. 5, p. 031410, 2018, doi: 10.1117/1.JMI.5.3.031410.

PSIOT commented 2 hours ago

Thank for you support Mr. Gallas.