search

ucl-pond / pySuStaIn

Subtype and Stage Inference (SuStaIn) algorithm with an example using simulated data.
MIT License
130 stars 63 forks source link

Add files via upload #45

Open Seymour22 opened 1 year ago

Seymour22 commented 1 year ago

Added code to print statistical difference between cosecutive models with n+1 subtypes

noxtoby commented 1 year ago

Had a quick look on my phone. Is this a t test between BIC? Are you sure that this is valid?

Seymour22 commented 1 year ago

Yes, this is the t test between BIC. I've checked it on the simulated data which supports ground truth of two subtypes and not three subtypes. See "SuStaIn tutorial using simulated data updated stats between models" file in notebooks

sea-shunned commented 1 year ago

I think Neil's comment is more about how meaningful a t-test between BIC is. That justification is key, and I think it'd be prudent to have that rather than integrate this based on empirical results in the simulated data.

From a code perspective, if we were to do this, you should be looping over all pairs of subtypes, rather than the for loop currently implemented. After this, there should also be the option for multiple comparison correction.

Seymour22 commented 1 year ago

Thanks for clarifying the issue @sea-shunned. I guess increasing the sample size for the t-test i.e number of cross-validations wouldn't work either as we'd end up with similar BIC values for each of the cross-validations. I think a simplier method is to compare the difference of BICs for each of the models trained on the main dataset. I'll update the code in another notebook.