Open qigongFDA opened 8 years ago
Create a new class validateFunction in roemetz.core package to do the validation. validateFunction could call "simulation", "numerical", and "export" buttons functions to do the analysis and export. In order to call these functions from other class, we move out them from button listener.
Create a variable doValidation in Roemetz.java class to avoid display windows during validation.
Previous, both simulation and numerical are working in parallel. This will make a problem in validation. In validation, program will try do export result before parallel analysis done. Thus, we have to figure out a method to wait analysis done before export result.
Good comments here.
From: Qi Gong [mailto:notifications@github.com] Sent: Wednesday, January 13, 2016 5:57 PM To: DIDSR/iMRMC Subject: Re: [iMRMC] Validate Split-Plot analysis (#81)
Create a new class validateFunction in roemetz.core package to do the validation. validateFunction could call "simulation", "numerical", and "export" buttons functions to do the analysis and export. In order to call these functions from other class, we move out them from button listener.
Create a a variable doValidation in Roemetz.java class to avoid display windows during validation.
Previous, both simulation and numerical are working in parallel. This will make a problem in validation. In validation, program will try do export result before parallel analysis done. Thus, we have to figure out a method to wait analysis done before export result.
— Reply to this email directly or view it on GitHubhttps://github.com/DIDSR/iMRMC/issues/81#issuecomment-171464156.
Can we set the validation to use the MLE variance estimate? I would like to characterize that too.
Here's the resources from our other project that you should use for this project.
https://github.com/DIDSR/mitoticCountSimulation/tree/bdg/src/000output/SPIEMI2016
Thanks!
Export reuslt in “(U-statistics, Not MLE) / (MLE, Not U-statistics)” format.
Enable MLE analysis in iRoemetz simulation analysis. But still have problem with MC variance and Numerical analysis.
Next steps for validation.
MLE should affect MC variance.
We might need add new variable varAUC_A and varAUC_B in DBRecord to record readers's AUC variance in each modality.
simulationOutputSummary.csv should has 12 colums: 1-6: mcMean of ACU_A, AUC_B, AUC_AminusB, varAUC_A, varAUC_B, varAUC_AminusB, 7-12 mcVar of ACU_A, AUC_B, AUC_AminusB, varAUC_A, varAUC_B, varAUC_AminusB,
numericalOutputSummary.csv should has 6 colums: ACU_A, AUC_B, AUC_AminusB, varAUC_A, varAUC_B, varAUC_AminusB,
Copy the comment from issue 91 We could use swarm command to run iRoeMetz validation parallel on linux. (i.e. swarm doValidation.txt)
There are two things we need to pay attention:
In validation we should avoid software display any warning. Because warning needs user to click "OK" button. It will stop validation.
In validation, we are not only export mc simulation and numerical results. We also generate trialResultArray to save all trails simulation results, and export it to .csv file.
We should generate several plot by mc simulation and numerical output. The plot could help us analysis how configures affect mcmean, mcvar, RSE and reject.
You are close to resolving this issue. You should start to document how to run iRoeMetz in parallel. When ready, I will try and run iRoeMetz in parallel following the documentation (no one-on-one help from you). Then we can work together to improve the documentation and ultimately close this issue.
Here is the draft of manual for validation. Validation_manual.txt
Update simulation and numerical output file. Now, both of them export all analysis results.
Could we close this issue?
Add sample validation input file in iMRMC/validation/
Move validation tools to the iMRMC repository and document.
README.MD and READMEexample1.md files show how to run validation in Linux.
Do we need to put it in iMRMC or iRoeMetz manual? Both of them have description about command line running. I don't think we should put the validation instruction in the manual, as it is not useful for outside users.
Input file 12 x 7 x 3 Raw file: https://github.com/DIDSR/iMRMC/tree/gh-pages/000_iRoeMetz/inputFiles/RMconfigurations. (12) Number of readers: 6,7,8,9,10,11,12 (7) Study Design: fully crossed, Split-plot ng: 2, 3 (3)
Command line execution: Use command line run the software