search

topepo / caret

caret (Classification And Regression Training) R package that contains misc functions for training and plotting classification and regression models
http://topepo.github.io/caret/index.html
1.61k stars 632 forks source link

Error: Something is wrong; all the logLoss metric values are missing; Warning: In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo, : There were missing values in resampled performance measures. #779

Closed slfan2013 closed 6 years ago

slfan2013 commented 6 years ago

Several methods, e.g. "plor", "gcvEarth","bagEarthGCV","bag","blackboost" etc, return the same problem.

Error: Something is wrong; all the logLoss metric values are missing. Warning: 2: In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo, : There were missing values in resampled performance measures. Code:

data = iris
control_method = "cv"

control_number = 5

control_repeats = 1

control_summaryFunction = get("multiClassSummary")

Control <- trainControl(method = control_method, # use N-fold cross validation
                        number = control_number, # the number of folds
                        repeats = control_repeats,
                        returnData = FALSE,
                        returnResamp = 'none',
                        savePredictions = "final",
                        classProbs = TRUE, 
                        summaryFunction = control_summaryFunction,
                        allowParallel = T)

train_metric = "ROC"

train_tuneLength = 5
x = data[,c(1:4)]

library(doParallel)
cl = makeCluster(7)
    registerDoParallel(cl)

Train <- caret::train(x, y = data[,5], 
                     method = "polr", 
                     preProcess = c("center", "scale"), 
                     metric = train_metric,
                     trControl = Control,
                     tuneLength = train_tuneLength 
        )

stopCluster(cl)
    remove(cl)
    registerDoSEQ()

The output is 

Something is wrong; all the logLoss metric values are missing:
    logLoss         AUC          prAUC        Accuracy       Kappa        Mean_F1   
 Min.   : NA   Min.   :0.5   Min.   : NA   Min.   : NA   Min.   : NA   Min.   : NA  
 1st Qu.: NA   1st Qu.:0.5   1st Qu.: NA   1st Qu.: NA   1st Qu.: NA   1st Qu.: NA  
 Median : NA   Median :0.5   Median : NA   Median : NA   Median : NA   Median : NA  
 Mean   :NaN   Mean   :0.5   Mean   :NaN   Mean   :NaN   Mean   :NaN   Mean   :NaN  
 3rd Qu.: NA   3rd Qu.:0.5   3rd Qu.: NA   3rd Qu.: NA   3rd Qu.: NA   3rd Qu.: NA  
 Max.   : NA   Max.   :0.5   Max.   : NA   Max.   : NA   Max.   : NA   Max.   : NA  
 NA's   :5                   NA's   :5     NA's   :5     NA's   :5     NA's   :5    
 Mean_Sensitivity Mean_Specificity Mean_Pos_Pred_Value Mean_Neg_Pred_Value Mean_Precision
 Min.   : NA      Min.   : NA      Min.   : NA         Min.   : NA         Min.   : NA   
 1st Qu.: NA      1st Qu.: NA      1st Qu.: NA         1st Qu.: NA         1st Qu.: NA   
 Median : NA      Median : NA      Median : NA         Median : NA         Median : NA   
 Mean   :NaN      Mean   :NaN      Mean   :NaN         Mean   :NaN         Mean   :NaN   
 3rd Qu.: NA      3rd Qu.: NA      3rd Qu.: NA         3rd Qu.: NA         3rd Qu.: NA   
 Max.   : NA      Max.   : NA      Max.   : NA         Max.   : NA         Max.   : NA   
 NA's   :5        NA's   :5        NA's   :5           NA's   :5           NA's   :5     
  Mean_Recall  Mean_Detection_Rate Mean_Balanced_Accuracy
 Min.   : NA   Min.   : NA         Min.   : NA           
 1st Qu.: NA   1st Qu.: NA         1st Qu.: NA           
 Median : NA   Median : NA         Median : NA           
 Mean   :NaN   Mean   :NaN         Mean   :NaN           
 3rd Qu.: NA   3rd Qu.: NA         3rd Qu.: NA           
 Max.   : NA   Max.   : NA         Max.   : NA           
 NA's   :5     NA's   :5           NA's   :5   

Error: Stopping
In addition: Warning messages:
1: In train.default(x, y = data[, 5], method = train_method, preProcess = c("center",  :
  The metric "ROC" was not in the result set. logLoss will be used instead.
2: In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,  :
  There were missing values in resampled performance measures.

My sessionInfo is:

R version 3.4.2 (2017-09-28)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] rrlda_1.1          matrixcalc_1.0-3   glasso_1.8         mvoutlier_2.0.8    sgeostat_1.0-27   
 [6] pcaPP_1.9-72       doParallel_1.0.11  iterators_1.0.8    foreach_1.4.3      MLmetrics_1.1.1   
[11] caret_6.0-77.9000  ggplot2_2.2.1.9000 lattice_0.20-35   

loaded via a namespace (and not attached):
  [1] pacman_0.4.6          tidyselect_0.2.3      lme4_1.1-14           grid_3.4.2           
  [5] combinat_0.0-8        trimcluster_0.1-2     ranger_0.8.0          deepnet_0.2          
  [9] munsell_0.4.3         codetools_0.2-15      xgboost_0.6-4         sROC_0.1-2           
 [13] withr_2.1.0.9000      colorspace_1.3-2      sparseLDA_0.1-9       stats4_3.4.2         
 [17] ROCR_1.0-7            robustbase_0.92-8     vcd_1.4-3             dimRed_0.1.0         
 [21] rJava_0.9-9           VIM_4.7.0             kohonen_3.0.4         lars_1.2             
 [25] bnclassify_0.3.3      cvTools_0.3.2         mnormt_1.5-5          adabag_4.1           
 [29] TH.data_1.0-8         ipred_0.9-6           randomForest_4.6-12   party_1.2-3          
 [33] diptest_0.75-7        R6_2.2.2              rocc_1.2              robCompositions_2.0.6
 [37] VGAM_1.0-4            evtree_1.0-5          flexmix_2.3-14        DRR_0.0.2            
 [41] robustHD_0.5.1        bitops_1.0-6          reshape_0.8.7         assertthat_0.2.0     
 [45] scales_0.5.0.9000     multcomp_1.4-8        nnet_7.3-12           gtable_0.2.0         
 [49] ddalpha_1.3.1         mda_0.4-10            rFerns_2.0.2          sandwich_2.4-0       
 [53] timeDate_3012.100     rlang_0.1.4           MatrixModels_0.4-1    CVST_0.2-1           
 [57] RcppRoll_0.2.2        Rborist_0.1-8         inTrees_1.1           splines_3.4.2        
 [61] lazyeval_0.2.1        ModelMetrics_1.1.0    broom_0.4.2           rrcovHD_0.2-5        
 [65] reshape2_1.4.2        tools_3.4.2           lava_1.5.1            spls_2.2-1           
 [69] psych_1.7.8           gplots_3.0.1          RColorBrewer_1.1-2    Rsolnp_1.16          
 [73] Rcpp_0.12.13          plyr_1.8.4            purrr_0.2.4           rpart_4.1-11         
 [77] RSNNS_0.4-9           zoo_1.8-0             sfsmisc_1.1-1         cluster_2.0.6        
 [81] magrittr_1.5          data.table_1.10.4-3   SparseM_1.77          lmtest_0.9-35        
 [85] truncnorm_1.0-7       mvtnorm_1.0-6         RWeka_0.4-35          sda_1.3.7            
 [89] xtable_1.8-2          klaR_0.6-12           pbkrtest_0.4-7        mclust_5.3           
 [93] compiler_3.4.2        snn_1.1               bdsmatrix_1.3-2       tibble_1.3.4         
 [97] KernSmooth_2.23-15    minqa_1.2.4           sparsediscrim_0.2.4   entropy_1.2.1        
[101] mgcv_1.8-20           corpcor_1.6.9         Formula_1.2-2         tidyr_0.7.2          
[105] rrcov_1.4-3           lubridate_1.7.1       MASS_7.3-47           fpc_2.1-10           
[109] arules_1.5-4          boot_1.3-20           Matrix_1.2-11         car_2.1-5            
[113] C50_0.1.0-24          perry_0.2.0           gdata_2.18.0          mlbench_2.1-1        
[117] bindr_0.1             gower_0.1.2           pkgconfig_2.0.1       elasticnet_1.1       
[121] coin_1.2-1            foreign_0.8-69        laeken_0.4.6          sp_1.2-5             
[125] frbs_3.1-0            extraTrees_1.0.5      recipes_0.1.0.9000    RRF_1.7              
[129] RWekajars_3.9.1-4     prodlim_1.6.1         stringr_1.2.0         pls_2.6-0            
[133] strucchange_1.5-1     robustDA_1.1          kernlab_0.9-25        gtools_3.5.0         
[137] quantreg_5.34         modeltools_0.2-21     nloptr_1.0.4          sdwd_1.0.2           
[141] nlme_3.1-131          bindrcpp_0.2          GGally_1.3.2          DEoptimR_1.0-8       
[145] survival_2.41-3       vbmp_1.44.0           glue_1.2.0            fdrtool_1.2.15       
[149] gbm_2.1.3             prabclus_2.2-6        glmnet_2.0-13         class_7.3-14         
[153] stringi_1.1.5         partykit_1.1-1        caTools_1.17.1        dplyr_0.7.4          
[157] e1071_1.6-8
topepo commented 6 years ago

These are resolved in the devel version. A version is going to CRAN this week.

slfan2013 commented 6 years ago

Thank you Max. If I download the developed version of caret from github, will the problem be solved?

topepo commented 6 years ago

Yes (and if you still see issues, post them here).

You can get the development version using

library(devtools)
install_github("topepo/caret/pkg/caret")

but you would have to be able to compile from source. There is a compiled version here but only for another day or so.

slfan2013 commented 6 years ago

Hi @topepo , Seems that I still got same error after installing github caret_6.0-78. For example, using bagEarthGCV.

data = iris

control_method = "cv"

control_number = 5

control_repeats = 1

control_summaryFunction = get("multiClassSummary")

Control1 <- trainControl(method = control_method, # use N-fold cross validation
                        number = control_number, # the number of folds
                        repeats = control_repeats,
                        returnData = FALSE,
                        returnResamp = 'none',
                        savePredictions = "final",
                        classProbs = TRUE, 
                        summaryFunction = control_summaryFunction,
                        allowParallel = T)
Control2 <- trainControl(method = control_method, # use N-fold cross validation
                         number = control_number, # the number of folds
                         repeats = control_repeats,
                         returnData = FALSE,
                         returnResamp = 'none',
                         savePredictions = "final",
                         classProbs = TRUE, 
                         summaryFunction = control_summaryFunction,
                         allowParallel = T)

train_method = as.character(model_info$model[3])
train_metric1 = "ROC"
train_metric2 = "Accuracy"
train_tuneLength = 5
x = data[,c(1:4)]

library(doParallel)

caret::train(x, y = data[,5], 
                     method = "bagEarthGCV", 
                     preProcess = c("center", "scale"), 
                     metric = train_metric2,
                     trControl = Control2,
                     tuneLength = train_tuneLength 
        )

Error:


Something is wrong; all the logLoss metric values are missing:
    logLoss         AUC          prAUC        Accuracy       Kappa        Mean_F1   
 Min.   : NA   Min.   :0.5   Min.   : NA   Min.   : NA   Min.   : NA   Min.   : NA  
 1st Qu.: NA   1st Qu.:0.5   1st Qu.: NA   1st Qu.: NA   1st Qu.: NA   1st Qu.: NA  
 Median : NA   Median :0.5   Median : NA   Median : NA   Median : NA   Median : NA  
 Mean   :NaN   Mean   :0.5   Mean   :NaN   Mean   :NaN   Mean   :NaN   Mean   :NaN  
 3rd Qu.: NA   3rd Qu.:0.5   3rd Qu.: NA   3rd Qu.: NA   3rd Qu.: NA   3rd Qu.: NA  
 Max.   : NA   Max.   :0.5   Max.   : NA   Max.   : NA   Max.   : NA   Max.   : NA  
 NA's   :1                   NA's   :1     NA's   :1     NA's   :1     NA's   :1    
 Mean_Sensitivity Mean_Specificity Mean_Pos_Pred_Value Mean_Neg_Pred_Value Mean_Precision
 Min.   : NA      Min.   : NA      Min.   : NA         Min.   : NA         Min.   : NA   
 1st Qu.: NA      1st Qu.: NA      1st Qu.: NA         1st Qu.: NA         1st Qu.: NA   
 Median : NA      Median : NA      Median : NA         Median : NA         Median : NA   
 Mean   :NaN      Mean   :NaN      Mean   :NaN         Mean   :NaN         Mean   :NaN   
 3rd Qu.: NA      3rd Qu.: NA      3rd Qu.: NA         3rd Qu.: NA         3rd Qu.: NA   
 Max.   : NA      Max.   : NA      Max.   : NA         Max.   : NA         Max.   : NA   
 NA's   :1        NA's   :1        NA's   :1           NA's   :1           NA's   :1     
  Mean_Recall  Mean_Detection_Rate Mean_Balanced_Accuracy
 Min.   : NA   Min.   : NA         Min.   : NA           
 1st Qu.: NA   1st Qu.: NA         1st Qu.: NA           
 Median : NA   Median : NA         Median : NA           
 Mean   :NaN   Mean   :NaN         Mean   :NaN           
 3rd Qu.: NA   3rd Qu.: NA         3rd Qu.: NA           
 Max.   : NA   Max.   : NA         Max.   : NA           
 NA's   :1     NA's   :1           NA's   :1             
Error: Stopping
In addition: Warning messages:
1: In train.default(x, y = data[, 5], method = train_method, preProcess = c("center",  :
  The metric "ROC" was not in the result set. logLoss will be used instead.
2: model fit failed for Fold1: degree=1 Error in get(ctr, mode = "function", envir = parent.frame()) : 
  object 'contr.earth.response' of mode 'function' was not found

3: model fit failed for Fold2: degree=1 Error in get(ctr, mode = "function", envir = parent.frame()) : 
  object 'contr.earth.response' of mode 'function' was not found

4: model fit failed for Fold3: degree=1 Error in get(ctr, mode = "function", envir = parent.frame()) : 
  object 'contr.earth.response' of mode 'function' was not found

5: model fit failed for Fold4: degree=1 Error in get(ctr, mode = "function", envir = parent.frame()) : 
  object 'contr.earth.response' of mode 'function' was not found

6: model fit failed for Fold5: degree=1 Error in get(ctr, mode = "function", envir = parent.frame()) : 
  object 'contr.earth.response' of mode 'function' was not found

7: In nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo,  :
  There were missing values in resampled performance measures.

My sessionInfo is,

R version 3.4.2 (2017-09-28)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    

attached base packages:
 [1] parallel  stats4    grid      stats     graphics  grDevices utils     datasets  methods  
[10] base     

other attached packages:
 [1] doParallel_1.0.11  iterators_1.0.8    foreach_1.4.3      wsrf_1.7.17        Rcpp_0.12.13      
 [6] plyr_1.8.4         party_1.2-3        strucchange_1.5-1  sandwich_2.4-0     zoo_1.8-0         
[11] modeltools_0.2-21  mvtnorm_1.0-6      MLmetrics_1.1.1    caret_6.0-78       ggplot2_2.2.1.9000
[16] lattice_0.20-35    devtools_1.13.4   

loaded via a namespace (and not attached):
 [1] nlme_3.1-131        bitops_1.0-6        lubridate_1.7.1     dimRed_0.1.0       
 [5] httr_1.3.1          C50_0.1.0-24        tools_3.4.2         R6_2.2.2           
 [9] KernSmooth_2.23-15  rpart_4.1-11        lazyeval_0.2.1      colorspace_1.3-2   
[13] nnet_7.3-12         withr_2.1.0.9000    gbm_2.1.3           tidyselect_0.2.3   
[17] mnormt_1.5-5        klaR_0.6-12         curl_3.0            compiler_3.4.2     
[21] git2r_0.19.0        glmnet_2.0-13       pacman_0.4.6        caTools_1.17.1     
[25] scales_0.5.0.9000   sfsmisc_1.1-1       DEoptimR_1.0-8      psych_1.7.8        
[29] robustbase_0.92-8   plotmo_3.3.4        stringr_1.2.0       digest_0.6.12      
[33] foreign_0.8-69      pkgconfig_2.0.1     plotrix_3.6-6       RWekajars_3.9.1-4  
[37] rlang_0.1.4         ddalpha_1.3.1       VGAM_1.0-4          bindr_0.1          
[41] combinat_0.0-8      gtools_3.5.0        dplyr_0.7.4         ModelMetrics_1.1.0 
[45] magrittr_1.5        Formula_1.2-2       Matrix_1.2-11       munsell_0.4.3      
[49] RWeka_0.4-35        partykit_1.1-1      stringi_1.1.5       multcomp_1.4-8     
[53] rpartScore_1.0-1    MASS_7.3-47         gplots_3.0.1        recipes_0.1.1      
[57] gdata_2.18.0        earth_4.5.1         splines_3.4.2       knitr_1.17         
[61] xgboost_0.6-4       reshape2_1.4.2      codetools_0.2-15    CVST_0.2-1         
[65] glue_1.2.0          data.table_1.10.4-3 mlbench_2.1-1       bst_0.3-14         
[69] gtable_0.2.0        purrr_0.2.4         tidyr_0.7.2         kernlab_0.9-25     
[73] assertthat_0.2.0    TeachingDemos_2.10  DRR_0.0.2           gower_0.1.2        
[77] coin_1.2-1          prodlim_1.6.1       broom_0.4.2         e1071_1.6-8        
[81] class_7.3-14        survival_2.41-3     timeDate_3042.101   RcppRoll_0.2.2     
[85] tibble_1.3.4        rJava_0.9-9         memoise_1.1.0       bindrcpp_0.2       
[89] lava_1.5.1          RSNNS_0.4-9         TH.data_1.0-8       ROCR_1.0-7         
[93] ipred_0.9-6
topepo commented 6 years ago

I didn't realize that earth's generalized linear model code could use more than two classes. I'll look into this more now that I've done an initial update to these models.

topepo commented 6 years ago

Please try using these changes and verify that it solves the issue.

slfan2013 commented 6 years ago

Thank you @topepo .

The bagEarthGCV is working! However, blackboost is not working and same issue (all NA's). Let me run all the method and I'll let you know which methods are having this error.

Thanks again.

topepo commented 6 years ago

The warnings from blackboost say

family = Multinomial() only works with Kronecker prodcut base-learners, i.e., combined base-learners of the form bl1 %O% bl2 fitted via gamboost() or mboost(). See ‘?Multinomial’ for details.

I interpret this as saying that it doesn't work for these data.

topepo commented 6 years ago

Linear or additive multinomial logit models can be fitted using Multinomial(); although is family requires some extra effort for model specification (see example). More specifically, the predictor must be in the form of a linear array model (see %O%). Note that this family does not work with tree-based base-learners at the moment. The class corresponding to the last level of the factor coding of the response is used as reference class.