search

raamana / neuropredict

Easy and comprehensive assessment of predictive power, with support for neuroimaging features
https://raamana.github.io/neuropredict/
Apache License 2.0
99 stars 27 forks source link

Classification blocked with multiple 1-dimensional features #56

Closed mattvan83 closed 4 years ago

mattvan83 commented 4 years ago

Hello,

I have problems when using multiple 1-D features with the following command:

      neuropredict -m meta_data.csv \
      -d mask_WMpet_av45_early.ero1.25mm.csv mask_WMpet_av45_early.ero1.5mm.csv \
      -o outdir -t 0.8 -n 250 -k 'all' \
     --gs_level 'exhaustive' --classifier "LinearSVC"

The CV trial were well launched for the first subgroup CN,MCI for both 1-D features but it seemed that at the end of CV trials the process got stuck at this stage:

>

>  Python 3.7.3
> > SGE recognized, job set up with 40 slots.
> > Positive class inferred for AUC calculation: CN
> > Running neuropredict 0.5+34.g220af55.dirty
> > 
> > Requested features for analysis:
> > get_data_matrix from /netapp/vol2_agewell/pro/IMAP/imap_mvh/CAT12/pet/Analyses/ML/All/CN_vs_MCI+AD/av45_early/pons/start0_dur4/mask_WMpet_av45_early.ero1.25mm.csv
> > get_data_matrix from /netapp/vol2_agewell/pro/IMAP/imap_mvh/CAT12/pet/Analyses/ML/All/CN_vs_MCI+AD/av45_early/pons/start0_dur4/mask_WMpet_av45_early.ero1.5mm.csv
> > Ignoring imputation strategy chosen, as no missing data were found!
> > 
> > Data import is done.
> > 
> > 
> > Requested processing for the following subgroups:
> > DIS,CN
> > 
> > --------------------------------------------------------------------------------
> > Processing subgroup : DIS,CN (1/1)
> > --------------------------------------------------------------------------------
> > SGE recognized, job set up with 40 slots.
> > Training percentage      : 0.8
> > Number of CV repetitions : 250
> > Classifier chosen        : linearsvc
> > Feature selection chosen : variancethreshold
> > Level of grid search     : exhaustive
> > Number of processors     : 40
> > Saving the results to 
> >   /netapp/vol2_agewell/pro/IMAP/imap_mvh/CAT12/pet/Analyses/ML/All/CN_vs_MCI+AD/av45_early/pons/start0_dur4/linearsvc/binary_WMmasks/CN_DIS
> > 
> > -------------------------
> > All datasets contain:
> >  
> > 101 samples, 2 classes, 1 features
> > Class  CN : 71 samples
> > Class DIS : 30 samples
> > -------------------------
> > 
> > Estimated chance accuracy : 0.500
> > 
> > Different classes in the training set are stratified to match the smallest class!
> > Parallelizing the repetitions of CV with 40 processes ...
> > CV trial 52     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6525    weighted AUC: 0.7482
> > CV trial 78     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631    weighted AUC: 0.8227
> > CV trial 50     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5993    weighted AUC: 0.6596
> > CV trial 72     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6082    weighted AUC: 0.6596
> > CV trial 46     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252    weighted AUC: 0.7021
> > CV trial 58     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085    weighted AUC: 0.8475
> > CV trial 38     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933    weighted AUC: 0.7518
> > CV trial 16     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358    weighted AUC: 0.7801
> > CV trial 12     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660    weighted AUC: 0.8794
> > CV trial 0      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6738    weighted AUC: 0.6950
> > CV trial 68     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7057    weighted AUC: 0.8369
> > CV trial 14     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8794
> > CV trial 22     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677    weighted AUC: 0.8333
> > CV trial 6      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631    weighted AUC: 0.7234
> > CV trial 48     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571    weighted AUC: 0.8298
> > CV trial 10     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720    weighted AUC: 0.7624
> > CV trial 64     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085    weighted AUC: 0.8191
> > CV trial 20     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677    weighted AUC: 0.8688
> > CV trial 4      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191    weighted AUC: 0.8688
> > CV trial 56     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553    weighted AUC: 0.8262
> > CV trial 70     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191    weighted AUC: 0.8440
> > CV trial 74     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.7553
> > CV trial 36     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.8050
> > CV trial 2      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.9043
> > CV trial 54     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191    weighted AUC: 0.8830
> > CV trial 24     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447    weighted AUC: 0.9043
> > CV trial 76     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340    weighted AUC: 0.9220
> > CV trial 42     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553    weighted AUC: 0.8759
> > CV trial 26     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.9078
> > CV trial 62     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571    weighted AUC: 0.8103
> > CV trial 66     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6418    weighted AUC: 0.7624
> > CV trial 60     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8511    weighted AUC: 0.8972
> > CV trial 72     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6294    weighted AUC: 0.6418
> > CV trial 40     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660    weighted AUC: 0.9149
> > CV trial 52     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720    weighted AUC: 0.7340
> > CV trial 16     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.7411
> > CV trial 46     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720    weighted AUC: 0.6489
> > CV trial 38     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6507    weighted AUC: 0.7376
> > CV trial 78     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358    weighted AUC: 0.8014
> > CV trial 50     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6099    weighted AUC: 0.6454
> > CV trial 58     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191    weighted AUC: 0.8298
> > CV trial 6      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145    weighted AUC: 0.6879
> > CV trial 14     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.8688
> > CV trial 36     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6844    weighted AUC: 0.8156
> > CV trial 18     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.8014
> > CV trial 0      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5904    weighted AUC: 0.6454
> > CV trial 20     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571    weighted AUC: 0.8582
> > CV trial 68     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677    weighted AUC: 0.7979
> > CV trial 4      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8511    weighted AUC: 0.8440
> > CV trial 74     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660    weighted AUC: 0.7358
> > CV trial 56     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447    weighted AUC: 0.8050
> > CV trial 34     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5816    weighted AUC: 0.7518
> > CV trial 12     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447    weighted AUC: 0.8723
> > CV trial 66     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.7376
> > CV trial 70     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6844    weighted AUC: 0.8227
> > CV trial 64     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358    weighted AUC: 0.8121
> > CV trial 24     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.8794
> > CV trial 76     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234    weighted AUC: 0.9007
> > CV trial 79     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5904    weighted AUC: 0.7021
> > CV trial 53     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720    weighted AUC: 0.7766
> > CV trial 2      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.8936
> > CV trial 73     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.9255
> > CV trial 54     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571    weighted AUC: 0.8759
> > CV trial 62     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571    weighted AUC: 0.8014
> > CV trial 17     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358    weighted AUC: 0.7624
> > CV trial 22     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677    weighted AUC: 0.8191
> > CV trial 39     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631    weighted AUC: 0.7695
> > CV trial 60     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191    weighted AUC: 0.9007
> > CV trial 51     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8404    weighted AUC: 0.8227
> > CV trial 26     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.9007
> > CV trial 42     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.8546
> > CV trial 48     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571    weighted AUC: 0.8121
> > CV trial 59     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.8014
> > CV trial 69     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.7695
> > CV trial 47     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039    weighted AUC: 0.7695
> > CV trial 21     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553    weighted AUC: 0.9220
> > CV trial 28     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8316    weighted AUC: 0.9007
> > CV trial 37     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660    weighted AUC: 0.8475
> > CV trial 57     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660    weighted AUC: 0.8582
> > CV trial 1      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8440
> > CV trial 40     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447    weighted AUC: 0.9149
> > CV trial 30     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085    weighted AUC: 0.8582
> > CV trial 18     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358    weighted AUC: 0.7589
> > CV trial 65     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.7837
> > CV trial 75     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.8617
> > CV trial 25     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720    weighted AUC: 0.7943
> > CV trial 13     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.8121
> > CV trial 79     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525    weighted AUC: 0.6915
> > CV trial 61     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553    weighted AUC: 0.7766
> > CV trial 71     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8156
> > CV trial 32     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8316    weighted AUC: 0.9291
> > CV trial 3      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.8440
> > CV trial 53     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039    weighted AUC: 0.7411
> > CV trial 55     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.8298
> > CV trial 15     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933    weighted AUC: 0.8582
> > CV trial 23     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.7872
> > CV trial 34     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5284    weighted AUC: 0.7376
> > CV trial 59     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447    weighted AUC: 0.7713
> > CV trial 43     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.7766
> > CV trial 77     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252    weighted AUC: 0.7305
> > CV trial 49     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039    weighted AUC: 0.8528
> > CV trial 27     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.8191
> > CV trial 73     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340    weighted AUC: 0.8936
> > CV trial 39     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5798    weighted AUC: 0.7305
> > CV trial 47     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145    weighted AUC: 0.7234
> > CV trial 69     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145    weighted AUC: 0.7199
> > CV trial 21     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021    weighted AUC: 0.9149
> > CV trial 44     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677    weighted AUC: 0.8369
> > CV trial 65     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447    weighted AUC: 0.7943
> > CV trial 41     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826    weighted AUC: 0.7092
> > CV trial 80     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6011    weighted AUC: 0.7305
> > CV trial 51     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6738    weighted AUC: 0.8121
> > CV trial 57     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7128    weighted AUC: 0.8369
> > CV trial 75     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.8156
> > CV trial 1      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.8191
> > CV trial 63     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340    weighted AUC: 0.8582
> > CV trial 82     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.8475
> > CV trial 3      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.8050
> > CV trial 23     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039    weighted AUC: 0.7270
> > CV trial 61     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933    weighted AUC: 0.7553
> > CV trial 55     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340    weighted AUC: 0.8014
> > CV trial 43     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418    weighted AUC: 0.7270
> > CV trial 77     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145    weighted AUC: 0.7234
> > CV trial 25     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6613    weighted AUC: 0.7872
> > CV trial 84     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.9362
> > CV trial 30     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358    weighted AUC: 0.8333
> > CV trial 19     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298    weighted AUC: 0.8298
> > CV trial 37     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.8298
> > CV trial 88     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.8121
> > CV trial 96     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6507    weighted AUC: 0.7270
> > CV trial 86     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8759
> > CV trial 13     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.7837
> > CV trial 41     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206    weighted AUC: 0.6809
> > CV trial 5      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8617    weighted AUC: 0.9043
> > CV trial 102    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933    weighted AUC: 0.8333
> > CV trial 80     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011    weighted AUC: 0.7163
> > CV trial 92     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.9220
> > CV trial 15     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.8227
> > CV trial 104    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6436    weighted AUC: 0.7660
> > CV trial 35     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8333
> > CV trial 106    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5585    weighted AUC: 0.7163
> > CV trial 108    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6418    weighted AUC: 0.7163
> > CV trial 110    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.6844
> > CV trial 94     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8617    weighted AUC: 0.9255
> > CV trial 98     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447    weighted AUC: 0.8652
> > CV trial 82     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.8546
> > CV trial 100    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7890    weighted AUC: 0.8652
> > CV trial 49     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.8440
> > CV trial 63     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234    weighted AUC: 0.8475
> > CV trial 112    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.8475
> > CV trial 90     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8511    weighted AUC: 0.8511
> > CV trial 118    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6525    weighted AUC: 0.7411
> > CV trial 27     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.7943
> > CV trial 44     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571    weighted AUC: 0.8014
> > CV trial 124    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.8085
> > CV trial 84     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553    weighted AUC: 0.9255
> > CV trial 110    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4645    weighted AUC: 0.6454
> > CV trial 31     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.9149
> > CV trial 96     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401    weighted AUC: 0.7021
> > CV trial 108    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039    weighted AUC: 0.6738
> > CV trial 126    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8582
> > CV trial 81     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631    weighted AUC: 0.7589
> > CV trial 116    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085    weighted AUC: 0.9184
> > CV trial 104    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145    weighted AUC: 0.7163
> > CV trial 106    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5585    weighted AUC: 0.7163
> > CV trial 67     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7996    weighted AUC: 0.9007
> > CV trial 92     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.9043
> > CV trial 88     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145    weighted AUC: 0.7695
> > CV trial 120    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553    weighted AUC: 0.7695
> > CV trial 122    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358    weighted AUC: 0.8511
> > CV trial 130    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933    weighted AUC: 0.7376
> > CV trial 19     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298    weighted AUC: 0.8085
> > CV trial 102    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.8262
> > CV trial 98     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.8688
> > CV trial 112    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234    weighted AUC: 0.8298
> > CV trial 114    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571    weighted AUC: 0.8652
> > CV trial 86     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.8475
> > CV trial 35     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.8121
> > CV trial 100    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7057    weighted AUC: 0.8475
> > CV trial 71     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.7996
> > CV trial 118    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525    weighted AUC: 0.6986
> > CV trial 94     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784    weighted AUC: 0.8901
> > CV trial 83     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6738    weighted AUC: 0.8475
> > CV trial 128    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.8369
> > CV trial 85     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826    weighted AUC: 0.7695
> > CV trial 17     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6631    weighted AUC: 0.7447
> > CV trial 90     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8404    weighted AUC: 0.8511
> > CV trial 45     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8617    weighted AUC: 0.8191
> > CV trial 111    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6188    weighted AUC: 0.8333
> > CV trial 124    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933    weighted AUC: 0.7713
> > CV trial 97     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571    weighted AUC: 0.8582
> > CV trial 109    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8652
> > CV trial 105    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039    weighted AUC: 0.8369
> > CV trial 120    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660    weighted AUC: 0.7340
> > CV trial 113    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447    weighted AUC: 0.6702
> > CV trial 87     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660    weighted AUC: 0.9362
> > CV trial 81     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011    weighted AUC: 0.7305
> > CV trial 136    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5904    weighted AUC: 0.8050
> > CV trial 132    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6738    weighted AUC: 0.7589
> > CV trial 116    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.8901
> > CV trial 99     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298    weighted AUC: 0.8972
> > CV trial 119    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7021    weighted AUC: 0.8759
> > CV trial 103    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191    weighted AUC: 0.9113
> > CV trial 95     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.6879
> > CV trial 134    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720    weighted AUC: 0.8014
> > CV trial 130    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660    weighted AUC: 0.6986
> > CV trial 93     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677    weighted AUC: 0.8121
> > CV trial 125    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933    weighted AUC: 0.7482
> > CV trial 101    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7784    weighted AUC: 0.8652
> > CV trial 91     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7021    weighted AUC: 0.9255
> > CV trial 85     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039    weighted AUC: 0.7482
> > CV trial 128    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.8262
> > CV trial 107    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826    weighted AUC: 0.7801
> > CV trial 111    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6188    weighted AUC: 0.7979
> > CV trial 5      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784    weighted AUC: 0.8865
> > CV trial 31     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.9220
> > CV trial 138    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7784    weighted AUC: 0.8901
> > CV trial 114    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784    weighted AUC: 0.8333
> > CV trial 121    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8546
> > CV trial 113    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933    weighted AUC: 0.6312
> > CV trial 97     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.8546
> > CV trial 28     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8528    weighted AUC: 0.8617
> > CV trial 89     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6418    weighted AUC: 0.6702
> > CV trial 109    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.8688
> > CV trial 122    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660    weighted AUC: 0.8422
> > CV trial 87     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.9149
> > CV trial 105    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933    weighted AUC: 0.8191
> > CV trial 83     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571    weighted AUC: 0.8475
> > CV trial 95     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6826    weighted AUC: 0.6879
> > CV trial 140    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8546
> > CV trial 7      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191    weighted AUC: 0.8830
> > CV trial 125    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039    weighted AUC: 0.6844
> > CV trial 142    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085    weighted AUC: 0.8723
> > CV trial 136    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358    weighted AUC: 0.7908
> > CV trial 103    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.8759
> > CV trial 119    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021    weighted AUC: 0.8262
> > CV trial 99     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085    weighted AUC: 0.8759
> > CV trial 132    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6436    weighted AUC: 0.7234
> > CV trial 129    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.7021
> > CV trial 91     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6915    weighted AUC: 0.9184
> > CV trial 126    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.8493
> > CV trial 150    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660    weighted AUC: 0.7376
> > CV trial 93     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011    weighted AUC: 0.8014
> > CV trial 144    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.7872
> > CV trial 148    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.7323
> > CV trial 146    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.7624
> > CV trial 107    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6294    weighted AUC: 0.7411
> > CV trial 152    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447    weighted AUC: 0.8723
> > CV trial 101    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6950    weighted AUC: 0.8617
> > CV trial 115    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7234    weighted AUC: 0.9007
> > CV trial 154    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8759
> > CV trial 134    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401    weighted AUC: 0.8014
> > CV trial 29     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8582
> > CV trial 158    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7128    weighted AUC: 0.8121
> > CV trial 166    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340    weighted AUC: 0.7872
> > CV trial 137    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.7234
> > CV trial 168    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252    weighted AUC: 0.7057
> > CV trial 160    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191    weighted AUC: 0.8440
> > CV trial 123    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.8121
> > CV trial 117    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.8652
> > CV trial 164    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677    weighted AUC: 0.8369
> > CV trial 45     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6950    weighted AUC: 0.7979
> > CV trial 170    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6011    weighted AUC: 0.7766
> > CV trial 150    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447    weighted AUC: 0.7057
> > CV trial 7      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447    weighted AUC: 0.8723
> > CV trial 121    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.8050
> > CV trial 174    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7784    weighted AUC: 0.8298
> > CV trial 176    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7677    weighted AUC: 0.8369
> > CV trial 162    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826    weighted AUC: 0.7660
> > CV trial 156    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.7908
> > CV trial 127    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933    weighted AUC: 0.8014
> > CV trial 8      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8723    weighted AUC: 0.9326
> > CV trial 172    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6117    weighted AUC: 0.7589
> > CV trial 144    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553    weighted AUC: 0.7589
> > CV trial 148    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5691    weighted AUC: 0.7021
> > CV trial 129    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4965    weighted AUC: 0.6809
> > CV trial 180    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.8121
> > CV trial 140    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.8191
> > CV trial 146    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660    weighted AUC: 0.7624
> > CV trial 142    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191    weighted AUC: 0.8652
> > CV trial 154    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.8475
> > CV trial 115    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6915    weighted AUC: 0.8865
> > CV trial 29     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340    weighted AUC: 0.8227
> > CV trial 166    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021    weighted AUC: 0.7553
> > CV trial 89     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418    weighted AUC: 0.6348
> > CV trial 133    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.8298
> > CV trial 168    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.6702
> > CV trial 184    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5780    weighted AUC: 0.7589
> > CV trial 137    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418    weighted AUC: 0.6950
> > CV trial 158    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447    weighted AUC: 0.7837
> > CV trial 152    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7234    weighted AUC: 0.8262
> > CV trial 151    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5479    weighted AUC: 0.7305
> > CV trial 131    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191    weighted AUC: 0.9255
> > CV trial 160    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085    weighted AUC: 0.8227
> > CV trial 123    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358    weighted AUC: 0.7766
> > CV trial 32     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.0709
> > CV trial 135    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.7713
> > CV trial 170    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011    weighted AUC: 0.7270
> > CV trial 188    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.6454
> > CV trial 145    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7234    weighted AUC: 0.7553
> > CV trial 117    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677    weighted AUC: 0.8369
> > CV trial 156    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.7837
> > CV trial 127    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039    weighted AUC: 0.7890
> > CV trial 186    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826    weighted AUC: 0.7979
> > CV trial 178    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.7624
> > CV trial 141    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340    weighted AUC: 0.8511
> > CV trial 147    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039    weighted AUC: 0.8759
> > CV trial 180    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418    weighted AUC: 0.7979
> > CV trial 174    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784    weighted AUC: 0.8085
> > CV trial 149    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085    weighted AUC: 0.8582
> > CV trial 162    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6507    weighted AUC: 0.7163
> > CV trial 172    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5904    weighted AUC: 0.7270
> > CV trial 184    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5674    weighted AUC: 0.7340
> > CV trial 151    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6826    weighted AUC: 0.7163
> > CV trial 176    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7677    weighted AUC: 0.7872
> > CV trial 194    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6525    weighted AUC: 0.7092
> > CV trial 188    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4645    weighted AUC: 0.5993
> > CV trial 169    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7553    weighted AUC: 0.8688
> > CV trial 155    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571    weighted AUC: 0.7872
> > CV trial 171    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826    weighted AUC: 0.7766
> > CV trial 159    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6223    weighted AUC: 0.7482
> > CV trial 167    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.8511
> > CV trial 153    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.8652
> > CV trial 133    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7465    weighted AUC: 0.7766
> > CV trial 143    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660    weighted AUC: 0.8617
> > CV trial 198    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.8333
> > CV trial 161    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.8475
> > CV trial 202    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.6950
> > CV trial 182    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.8333
> > CV trial 164    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6844    weighted AUC: 0.8191
> > CV trial 186    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401    weighted AUC: 0.7411
> > CV trial 8      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8830    weighted AUC: 0.0674
> > CV trial 157    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447    weighted AUC: 0.9574
> > CV trial 141    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7021    weighted AUC: 0.8617
> > CV trial 135    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6631    weighted AUC: 0.7695
> > CV trial 145    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340    weighted AUC: 0.7199
> > CV trial 190    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039    weighted AUC: 0.7553
> > CV trial 181    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.7979
> > CV trial 196    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.7766
> > CV trial 33     feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8103    weighted AUC: 0.8936
> > CV trial 192    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8422    weighted AUC: 0.8546
> > CV trial 138    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7784    weighted AUC: 0.8901
> > CV trial 200    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085    weighted AUC: 0.9468
> > CV trial 149    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.8546
> > CV trial 173    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.8475
> > CV trial 177    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8014
> > CV trial 178    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7465    weighted AUC: 0.7340
> > CV trial 204    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8404
> > CV trial 206    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.7872
> > CV trial 163    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.8085
> > CV trial 185    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.8298
> > CV trial 171    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039    weighted AUC: 0.7801
> > CV trial 202    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206    weighted AUC: 0.6950
> > CV trial 169    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447    weighted AUC: 0.8440
> > CV trial 167    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.8369
> > CV trial 153    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447    weighted AUC: 0.8369
> > CV trial 131    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298    weighted AUC: 0.9326
> > CV trial 159    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6950    weighted AUC: 0.7340
> > CV trial 194    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6418    weighted AUC: 0.6879
> > CV trial 187    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571    weighted AUC: 0.7553
> > CV trial 155    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6738    weighted AUC: 0.7660
> > CV trial 212    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.8546
> > CV trial 198    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6826    weighted AUC: 0.8191
> > CV trial 189    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8794
> > CV trial 165    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.8652
> > CV trial 157    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7340    weighted AUC: 0.9397
> > CV trial 208    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7979    weighted AUC: 0.8546
> > CV trial 139    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720    weighted AUC: 0.7855
> > CV trial 190    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039    weighted AUC: 0.7305
> > CV trial 214    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5780    weighted AUC: 0.6879
> > CV trial 181    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7128    weighted AUC: 0.7660
> > CV trial 210    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.8121
> > CV trial 143    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553    weighted AUC: 0.8404
> > CV trial 9      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7039    weighted AUC: 0.8511
> > CV trial 196    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6738    weighted AUC: 0.7518
> > CV trial 182    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933    weighted AUC: 0.8227
> > CV trial 147    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039    weighted AUC: 0.8582
> > CV trial 175    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298    weighted AUC: 0.9113
> > CV trial 216    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252    weighted AUC: 0.7943
> > CV trial 173    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.8440
> > CV trial 177    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.7979
> > CV trial 206    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085    weighted AUC: 0.7518
> > CV trial 179    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7571    weighted AUC: 0.8121
> > CV trial 204    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085    weighted AUC: 0.8440
> > CV trial 218    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298    weighted AUC: 0.8759
> > CV trial 185    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660    weighted AUC: 0.8014
> > CV trial 226    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6312    weighted AUC: 0.8227
> > CV trial 224    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.8475
> > CV trial 220    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.7801
> > CV trial 195    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.7872
> > CV trial 200    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8511    weighted AUC: 0.9291
> > CV trial 163    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7465    weighted AUC: 0.7908
> > CV trial 161    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.8156
> > CV trial 230    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6401    weighted AUC: 0.8227
> > CV trial 199    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7447    weighted AUC: 0.8050
> > CV trial 212    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6613    weighted AUC: 0.8262
> > CV trial 187    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145    weighted AUC: 0.7447
> > CV trial 232    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6844    weighted AUC: 0.7837
> > CV trial 207    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.4752    weighted AUC: 0.6064
> > CV trial 33     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.0975
> > CV trial 228    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.8440
> > CV trial 203    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358    weighted AUC: 0.7766
> > CV trial 139    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525    weighted AUC: 0.7624
> > CV trial 183    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358    weighted AUC: 0.8262
> > CV trial 208    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.8404
> > CV trial 165    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.8191
> > CV trial 234    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298    weighted AUC: 0.9468
> > CV trial 214    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206    weighted AUC: 0.6631
> > CV trial 216    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039    weighted AUC: 0.7589
> > CV trial 191    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298    weighted AUC: 0.8511
> > CV trial 240    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660    weighted AUC: 0.8440
> > CV trial 236    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.8262
> > CV trial 210    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571    weighted AUC: 0.7943
> > CV trial 242    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298    weighted AUC: 0.8582
> > CV trial 224    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7553    weighted AUC: 0.7943
> > CV trial 189    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7979    weighted AUC: 0.8387
> > CV trial 220    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.7163
> > CV trial 213    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6950    weighted AUC: 0.8156
> > CV trial 238    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6844    weighted AUC: 0.8085
> > CV trial 218    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.8582
> > CV trial 230    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6401    weighted AUC: 0.7730
> > CV trial 226    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6312    weighted AUC: 0.8014
> > CV trial 179    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.8014
> > CV trial 201    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6507    weighted AUC: 0.8298
> > CV trial 207    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4645    weighted AUC: 0.5816
> > CV trial 246    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720    weighted AUC: 0.8121
> > CV trial 248    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6117    weighted AUC: 0.7163
> > CV trial 195    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.7518
> > CV trial 199    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.7837
> > CV trial 183    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6525    weighted AUC: 0.8298
> > CV trial 244    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358    weighted AUC: 0.8298
> > CV trial 203    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571    weighted AUC: 0.7270
> > CV trial 209    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7340    weighted AUC: 0.8511
> > CV trial 205    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5691    weighted AUC: 0.7801
> > CV trial 217    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085    weighted AUC: 0.9149
> > CV trial 240    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.8191
> > CV trial 221    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7660    weighted AUC: 0.9291
> > CV trial 225    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7358    weighted AUC: 0.8262
> > CV trial 232    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6436    weighted AUC: 0.7660
> > CV trial 211    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252    weighted AUC: 0.8262
> > CV trial 234    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298    weighted AUC: 0.9326
> > CV trial 236    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145    weighted AUC: 0.7943
> > CV trial 246    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720    weighted AUC: 0.7730
> > CV trial 213    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8404    weighted AUC: 0.8050
> > CV trial 209    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7447    weighted AUC: 0.8316
> > CV trial 219    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6631    weighted AUC: 0.7305
> > CV trial 248    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5071    weighted AUC: 0.7092
> > CV trial 201    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6613    weighted AUC: 0.7979
> > CV trial 67     feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7163    weighted AUC: 0.8617
> > CV trial 228    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933    weighted AUC: 0.8121
> > CV trial 9      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933    weighted AUC: 0.8262
> > CV trial 217    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085    weighted AUC: 0.8830
> > CV trial 215    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7766    weighted AUC: 0.9255
> > CV trial 244    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085    weighted AUC: 0.8121
> > CV trial 175    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298    weighted AUC: 0.8688
> > CV trial 231    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7145    weighted AUC: 0.7979
> > CV trial 227    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8085    weighted AUC: 0.8369
> > CV trial 241    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7465    weighted AUC: 0.8652
> > CV trial 211    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7571    weighted AUC: 0.8014
> > CV trial 205    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7039    weighted AUC: 0.7660
> > CV trial 235    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6933    weighted AUC: 0.8156
> > CV trial 191    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298    weighted AUC: 0.8475
> > CV trial 249    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6826    weighted AUC: 0.6773
> > CV trial 233    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6206    weighted AUC: 0.7234
> > CV trial 225    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7358    weighted AUC: 0.7908
> > CV trial 247    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7021    weighted AUC: 0.8865
> > CV trial 237    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7872    weighted AUC: 0.8652
> > CV trial 242    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8298    weighted AUC: 0.8333
> > CV trial 229    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6720    weighted AUC: 0.7872
> > CV trial 219    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5904    weighted AUC: 0.7199
> > CV trial 221    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660    weighted AUC: 0.8901
> > CV trial 238    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6011    weighted AUC: 0.7943
> > CV trial 245    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8298    weighted AUC: 0.8528
> > CV trial 222    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191    weighted AUC: 0.8794
> > CV trial 215    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660    weighted AUC: 0.9078
> > CV trial 249    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933    weighted AUC: 0.6312
> > CV trial 231    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7145    weighted AUC: 0.7447
> > CV trial 235    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6720    weighted AUC: 0.7943
> > CV trial 241    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191    weighted AUC: 0.8582
> > CV trial 247    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6915    weighted AUC: 0.8511
> > CV trial 229    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6933    weighted AUC: 0.7624
> > CV trial 233    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6206    weighted AUC: 0.6915
> > CV trial 237    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7872    weighted AUC: 0.8546
> > CV trial 227    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7660    weighted AUC: 0.8262
> > CV trial 239    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7252    weighted AUC: 0.8901
> > CV trial 245    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8404    weighted AUC: 0.8121
> > CV trial 243    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7996    weighted AUC: 0.8688
> > CV trial 222    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8085    weighted AUC: 0.8723
> > CV trial 239    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7252    weighted AUC: 0.8901
> > CV trial 192    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8316    weighted AUC: 0.8298
> > CV trial 197    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191    weighted AUC: 0.9539
> > CV trial 197    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7766    weighted AUC: 0.9433
> > CV trial 223    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.5709    weighted AUC: 0.7908
> > CV trial 243    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7890    weighted AUC: 0.8617
> > CV trial 193    feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.8191    weighted AUC: 0.9255
> > CV trial 223    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.4982    weighted AUC: 0.7943
> > CV trial 193    feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.8191    weighted AUC: 0.9397
>

If I launch the previous command line with only one 1-D feature as follows, everything is going well:

      neuropredict -m meta_data.csv \
      -d mask_WMpet_av45_early.ero1.25mm.csv \
      -o outdir -t 0.8 -n 250 -k 'all' \
      --gs_level 'exhaustive' --classifier "LinearSVC"

Do you have any idea?

Best,

Matthieu

Sources.zip

raamana commented 4 years ago

I don't see an error msg here - post the very end of the output also?

also, for n=100 samples, you don't need 250 repetitions. 50 should be fine.

for debugging purposes, just run it with 10 reps, and see if it finishes?

mattvan83 commented 4 years ago

The problem is that nothing comes after what what I outputed. All repetitions have run but the program seems to be blocked thereafter.

raamana commented 4 years ago

that's very weird - I will try look into this over the weekend

raamana commented 4 years ago

Hi Matthieu,

I ran neuropredict with your 1D feature sets on my laptop with the latest version (from #51), and it completed fine - see below.. Obviously there most things are different here (model, version etc), but I don't think 1D features are the problem. Try random forests model with only 10 reps?

(base) $ 17:37:15 SQuark-3 Sources >>  neuropredict -m meta_data.csv -d mask_WMpet_av45_early.ero1.* -n 10 -c 1

neuropredict version 0.6.dev3+21.gee9da15.dirty for Classification

Positive class inferred for AUC calculation: DIS

Requested features for analysis:
get_data_matrix from /Users/Reddy/Downloads/Sources/mask_WMpet_av45_early.ero1.25mm.csv
get_data_matrix from /Users/Reddy/Downloads/Sources/mask_WMpet_av45_early.ero1.5mm.csv

Data import is done.

Requested processing for the following subgroups:
CN,DIS

--------------------------------------------------------------------------------
Processing subgroup : CN,DIS (1/1)
--------------------------------------------------------------------------------
CN_DIS:
    101 samples, 2 modalities, dims: [1, 1]
    Identifiers: 0, 1
    Attributes:
    Classes n=2, sizes CN: 71, DIS: 30

Ignoring imputation strategy chosen, as no missing data were found!

CURRENT EXPERIMENT:
--------------------------------------------------
Training percentage      : 0.5
Number of CV repetitions : 10
Number of processors     : 1
Dim reduction method     : variancethreshold
Dim reduction size       : tenth
Predictive model chosen  : randomforestclassifier
Grid search level        : light

Estimated chance accuracy : 0.500

Saving results to:
 /Users/Reddy/Downloads/Sources/neuropredict_results/CN_DIS

CV run 0   dataset 0  :        accuracy_score    0.667  balanced_accuracy_score    0.529
CV run 0   dataset 1  :        accuracy_score    0.627  balanced_accuracy_score    0.529
CV run 1   dataset 0  :        accuracy_score    0.784  balanced_accuracy_score    0.690
CV run 1   dataset 1  :        accuracy_score    0.725  balanced_accuracy_score    0.630
CV run 2   dataset 0  :        accuracy_score    0.529  balanced_accuracy_score    0.670
CV run 2   dataset 1  :        accuracy_score    0.549  balanced_accuracy_score    0.682
CV run 3   dataset 0  :        accuracy_score    0.725  balanced_accuracy_score    0.669
CV run 3   dataset 1  :        accuracy_score    0.667  balanced_accuracy_score    0.647
CV run 4   dataset 0  :        accuracy_score    0.765  balanced_accuracy_score    0.682
CV run 4   dataset 1  :        accuracy_score    0.745  balanced_accuracy_score    0.691
CV run 5   dataset 0  :        accuracy_score    0.706  balanced_accuracy_score    0.603
CV run 5   dataset 1  :        accuracy_score    0.686  balanced_accuracy_score    0.548
CV run 6   dataset 0  :        accuracy_score    0.725  balanced_accuracy_score    0.619
CV run 6   dataset 1  :        accuracy_score    0.647  balanced_accuracy_score    0.625
CV run 7   dataset 0  :        accuracy_score    0.725  balanced_accuracy_score    0.678
CV run 7   dataset 1  :        accuracy_score    0.784  balanced_accuracy_score    0.718
CV run 8   dataset 0  :        accuracy_score    0.706  balanced_accuracy_score    0.575
CV run 8   dataset 1  :        accuracy_score    0.686  balanced_accuracy_score    0.562
CV run 9   dataset 0  :        accuracy_score    0.725  balanced_accuracy_score    0.691
CV run 9   dataset 1  :        accuracy_score    0.706  balanced_accuracy_score    0.676

Results saved to /Users/Reddy/Downloads/Sources/neuropredict_results/CN_DIS/results_neuropredict.pkl

Error in removing temp dir - remove it yourself:
{} /Users/Reddy/Downloads/Sources/neuropredict_results/CN_DIS/temp_dump
Metrics : accuracy_score, balanced_accuracy_score, area_under_roc
 # runs : 20, # datasets : 2

accuracy_score
     0  : median 0.7255  SD 0.0662
     1  : median 0.6863  SD 0.0625

balanced_accuracy_score
     0  : median 0.6695  SD 0.0532
     1  : median 0.6388  SD 0.0618

area_under_roc
     0  : median 0.7272  SD 0.0510
     1  : median 0.6954  SD 0.0715

Results have been saved to
    /Users/Reddy/Downloads/Sources/neuropredict_results/CN_DIS/results_neuropredict.pkl
All done.
raamana commented 4 years ago

Also, using the latest stable release from pypi 0.5.2 also works (after the fix for get_data_matrix()).. I'd say something in your setup isn't right.

(base) $ 18:20:58 SQuark-3 Sources >>  neuropredict -m meta_data.csv -d mask_WMpet_av45_early.ero1.* -n 10 -c 1
Positive class inferred for AUC calculation: CN
Running neuropredict 0.5.2

Requested features for analysis:
get_data_matrix from /Users/Reddy/Downloads/Sources/mask_WMpet_av45_early.ero1.25mm.csv
get_data_matrix from /Users/Reddy/Downloads/Sources/mask_WMpet_av45_early.ero1.5mm.csv
Ignoring imputation strategy chosen, as no missing data were found!

Data import is done.

Requested processing for the following subgroups:
DIS,CN

--------------------------------------------------------------------------------
Processing subgroup : DIS,CN (1/1)
--------------------------------------------------------------------------------
Training percentage      : 0.5
Number of CV repetitions : 10
Classifier chosen        : randomforestclassifier
Feature selection chosen : variancethreshold
Level of grid search     : light
Number of processors     : 1
Saving the results to
  /Users/Reddy/Downloads/Sources/neuropredict_results/CN_DIS

-------------------------
All datasets contain:

101 samples, 2 classes, 1 features
Class  CN : 71 samples
Class DIS : 30 samples
-------------------------

Estimated chance accuracy : 0.500

Different classes in the training set are stratified to match the smallest class!
CV trial 0      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7369    weighted AUC: 0.7542
CV trial 0      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7458    weighted AUC: 0.8024
CV trial 1      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7702    weighted AUC: 0.8173
CV trial 1      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7768    weighted AUC: 0.7911
CV trial 2      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7702    weighted AUC: 0.7601
CV trial 2      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6256    weighted AUC: 0.6708
CV trial 3      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7304    weighted AUC: 0.7423
CV trial 3      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6815    weighted AUC: 0.7631
CV trial 4      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7637    weighted AUC: 0.7274
CV trial 4      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.5190    weighted AUC: 0.5958
CV trial 5      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7637    weighted AUC: 0.8030
CV trial 5      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7458    weighted AUC: 0.7815
CV trial 6      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.7256    weighted AUC: 0.7357
CV trial 6      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.7167    weighted AUC: 0.6839
CV trial 7      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6750    weighted AUC: 0.6756
CV trial 7      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6262    weighted AUC: 0.5863
CV trial 8      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6905    weighted AUC: 0.6500
CV trial 8      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6435    weighted AUC: 0.6685
CV trial 9      feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy: 0.6637    weighted AUC: 0.6869
CV trial 9      feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy: 0.6548    weighted AUC: 0.6827

The most frequently selected parameter values are:
  For feature set 0 : mask_WMpet_av45_early.ero1.25mm
    random_forest_clf__criterion          : gini
    random_forest_clf__max_features       : sqrt
    random_forest_clf__min_samples_leaf   : 1
    random_forest_clf__n_estimators       : 50

  For feature set 1 : mask_WMpet_av45_early.ero1.5mm
    random_forest_clf__criterion          : gini
    random_forest_clf__max_features       : sqrt
    random_forest_clf__min_samples_leaf   : 1
    random_forest_clf__n_estimators       : 50

Exporting accuracy distribution                    .. Done.
Exporting confusion matrices                       .. Done.
Exporting misclassfiication rates                  .. Done.
Exporting feature importance values                .. Done.
Exporting subject-wise misclf frequencies          .. Done.

Median performance summary:
feature 0 mask_WMpet_av45_early.ero1.25mm : balanced accuracy 0.73   AUC 0.74
feature 1  mask_WMpet_av45_early.ero1.5mm : balanced accuracy 0.67   AUC 0.68

Saving the visualizations to
/Users/Reddy/Downloads/Sources/neuropredict_results
raamana commented 4 years ago

reopen if you still find it as an issue.