selbal.cv prints thousands of warning messages to the console

[mmp3]

Thank you for the great algorithm, well-functioning R package, and thorough documentation with examples! It has been a pleasure to use selbal!

Unfortunately, selbal.cv writes thousands of warning messages to the console output.

For example, following the example dataset from the vignette:

# Define x, y and z
  x <- HIV[,1:60]
  y <- HIV[,62]
  z <- data.frame(MSM = HIV[,61])

# Run selbal.cv function (with the default values for zero.rep and opt.cri)
  CV.BAL.dic <- selbal.cv(x = x, y = y, n.fold = 5, n.iter = 10,
                          covar = z, logit.acc = "AUC")

results in thousands of lines being printed to the console like this:

Setting direction: controls < cases
Setting direction: controls < cases
Setting direction: controls < cases
Setting levels: control = 0, case = 1
Setting direction: controls < cases
Setting levels: control = 0, case = 1
Setting levels: control = 0, case = 1
Setting levels: control = 0, case = 1
Setting levels: control = 0, case = 1
Setting levels: control = 0, case = 1
Setting direction: controls < cases
Setting direction: controls < cases
Setting direction: controls < cases
Setting direction: controls < cases
Setting direction: controls < cases
Setting levels: control = 0, case = 1
Setting direction: controls < cases
Setting levels: control = 0, case = 1
Setting levels: control = 0, case = 1
Setting levels: control = 0, case = 1
Setting levels: control = 0, case = 1
Setting direction: controls < cases

Looking around google, it seems that these messages are generated from pROC::roc.

Perhaps the argument quiet = TRUE should be provided to roc at lines 516 and 1940 in Selbal_Functions.R

[UVic-omics]

Hi @mmp3 !

First of all, thank you for using selbal. Running the vignette example you say, what I got is the following:

############################################################### 
 STARTING selbal.cv FUNCTION 
###############################################################

#-------------------------------------------------------------# 
# ZERO REPLACEMENT . . .

Loading required package: MASS
Loading required package: NADA
Loading required package: survival

Attaching package: ‘NADA’

The following object is masked from ‘package:stats’:

    cor

Loading required package: truncnorm

, . . . FINISHED. 
#-------------------------------------------------------------#

#-------------------------------------------------------------# 
# Starting the cross - validation procedure . . .
 . . . finished. 
#-------------------------------------------------------------# 
###############################################################

 The optimal number of variables is: 2 

Attaching package: ‘gridExtra’

The following object is masked from ‘package:Biobase’:

    combine

The following object is masked from ‘package:BiocGenerics’:

    combine

############################################################### 
 . . . FINISHED. 
###############################################################Warning messages:
1: package ‘zCompositions’ was built under R version 3.5.2 
2: package ‘pROC’ was built under R version 3.5.2 
3: package ‘doParallel’ was built under R version 3.5.2 
4: In e$fun(obj, substitute(ex), parent.frame(), e$data) :
  already exporting variable(s): logit.acc
5: package ‘ggplot2’ was built under R version 3.5.2 

So, I did not get any messages as you wrote.

[mmp3]

Hi @UVic-omics , Thank you for looking into this.

You do not see the warnings that I reported above because you are running an old version of R and an old version of package pROC.

This is proven by the NEWS for package pROC, which has the following line from June 2019 that exactly explains the warnings messages that I reported in the first post:

1.15.0 (2019-06-01)
  * 'roc' now prints messages when autodetecting 'levels' and 'direction' by default. Turn off with 'quiet = TRUE' or set these values explicitly

Therefore, if you update R and pROC, you will see the same warnings that I do.

For reference, here is the sessionInfo() for the first post above:

> sessionInfo()

R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.1 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] grid      parallel  stats     graphics  grDevices utils     datasets
[8] methods   base

other attached packages:
 [1] gtable_0.3.0        gridExtra_2.3       ggplot2_3.3.3
 [4] doParallel_1.0.16   iterators_1.0.13    foreach_1.5.1
 [7] pROC_1.17.0.1       Biobase_2.50.0      BiocGenerics_0.36.0
[10] zCompositions_1.3.4 truncnorm_1.0-8     NADA_1.6-1.1
[13] survival_3.2-7      MASS_7.3-53         plyr_1.8.6
[16] selbal_0.1.0        stringr_1.4.0       data.table_1.14.0

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.6        pillar_1.6.0      compiler_4.0.3    R.methodsS3_1.8.1
 [5] R.utils_2.10.1    tools_4.0.3       digest_0.6.27     bit_4.0.4
 [9] lifecycle_1.0.0   tibble_3.1.0      lattice_0.20-41   pkgconfig_2.0.3
[13] rlang_0.4.10      Matrix_1.3-2      withr_2.4.1       dplyr_1.0.2
[17] generics_0.1.0    vctrs_0.3.7       tidyselect_1.1.0  bit64_4.0.5
[21] glue_1.4.2        R6_2.5.0          fansi_0.4.2       farver_2.1.0
[25] purrr_0.3.4       magrittr_2.0.1    scales_1.1.1      codetools_0.2-18
[29] ellipsis_0.3.1    splines_4.0.3     colorspace_2.0-0  labeling_0.4.2
[33] utf8_1.2.1        stringi_1.5.3     munsell_0.5.0     crayon_1.4.1
[37] R.oo_1.24.0

[UVic-omics]

Ok, thank you so much @mmp3.

We take into account your recomendation of quiet=TRUE in order not have so many warnings.

Thank you!

[malucalle]

Hi @mmp3, We have updated the package according to your suggestion. Thank you!

[mmp3]

Glad I could help. Thank you, @malucalle !

[mmp3]

Hello again, I have updated the package to the latest version in github:

devtools::install_github(repo = "malucalle/selbal")

but the thousands of warning messages are still printed...

[malucalle]

Hi @mmp3, We fixed the problem. Not only roc() function but also auc() required quiet = TRUE Thanks!

[mmp3]

Yes, it is indeed fixed now. Thank you!