message 'Environment Modules is not available.' even though system("modulecmd bash -V") reports the correct version installed

[FelixErnst]

Hi,

I got a bit of a problem using systemPipeR. I always get the message, that 'Environment Modules is not available.' running runCommandline. However, the command to check this returns the correct version of the modules installed.

system("modulecmd bash -V")
#> Modules Release 4.2.2 (2019-02-17)

Did I miss something? Can you elaborate which influence this has, since the commands get run nonetheless?

SessionInfo

#> R version 3.6.2 (2019-12-12)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Debian GNU/Linux 10 (buster)
#> 
#> Matrix products: default
#> BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.3.5.so
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> attached base packages:
#> [1] stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> loaded via a namespace (and not attached):
#>  [1] compiler_3.6.2  magrittr_1.5    tools_3.6.2     htmltools_0.4.0
#>  [5] yaml_2.2.1      Rcpp_1.0.3      stringi_1.4.6   rmarkdown_2.1  
#>  [9] highr_0.8       knitr_1.28      stringr_1.4.0   xfun_0.12      
#> [13] digest_0.6.25   rlang_0.4.5     evaluate_0.14

[dcassol]

Hi @FelixErnst, Thank you for reporting this. Could you please let me know what version of systemPipeR you have installed?

This message is generated because internally runCommandline checks if you are on the system with Module Environment installed or not. If it is not available, you need to make sure to configure the PATH environment accordingly, for example, on your local machine. If you are running systemPipeR on the HPC with module available, internally, we use module load for all the software-defined on the param file.

Could you also try this and let me know what the results are?

system("modulecmd bash -V", ignore.stderr = TRUE, ignore.stdout = TRUE)!=1
system("declare -f module")
systemPipeR::module("list")

Thank you very much!

[FelixErnst]

Hi @dcassol,

Thanks for looking into this.

This is the output of the commands as requested. systemPipeR is version 1.20.0.

library(systemPipeR)
system("modulecmd bash -V", ignore.stderr = TRUE, ignore.stdout = TRUE)!=1
#> [1] TRUE
system("declare -f module")
#> sh: 1: declare: not found
#> Warning message:
#> In system("declare -f module") : error in running command
systemPipeR::module("list")
#> No Modulefiles Currently Loaded.

system("modulecmd bash -V", ignore.stderr = TRUE, ignore.stdout = TRUE) returns NULL, which or course is !=1. declare works on the command line using bash, but not from within R using system.

SessionInfo

#> R version 3.6.2 (2019-12-12)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Debian GNU/Linux 10 (buster)
#> 
#> Matrix products: default
#> BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.3.5.so
#> 
#> locale:
#>  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
#>  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
#>  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
#>  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
#>  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
#> 
#> attached base packages:
#> [1] stats4    parallel  stats     graphics  grDevices utils     datasets 
#> [8] methods   base     
#> 
#> other attached packages:
#>  [1] systemPipeR_1.20.0          ShortRead_1.44.3           
#>  [3] GenomicAlignments_1.22.1    SummarizedExperiment_1.16.1
#>  [5] DelayedArray_0.12.2         matrixStats_0.55.0         
#>  [7] Biobase_2.46.0              BiocParallel_1.20.1        
#>  [9] Rsamtools_2.2.3             Biostrings_2.54.0          
#> [11] XVector_0.26.0              GenomicRanges_1.38.0       
#> [13] GenomeInfoDb_1.22.0         IRanges_2.20.2             
#> [15] S4Vectors_0.24.3            BiocGenerics_0.32.0        
#> 
#> loaded via a namespace (and not attached):
#>  [1] Category_2.52.1          bitops_1.0-6             bit64_0.9-7             
#>  [4] RColorBrewer_1.1-2       progress_1.2.2           httr_1.4.1              
#>  [7] Rgraphviz_2.30.0         backports_1.1.5          tools_3.6.2             
#> [10] R6_2.4.1                 DBI_1.1.0                colorspace_1.4-1        
#> [13] withr_2.1.2              tidyselect_1.0.0         prettyunits_1.1.1       
#> [16] bit_1.1-15.2             curl_4.3                 compiler_3.6.2          
#> [19] graph_1.64.0             rtracklayer_1.46.0       checkmate_2.0.0         
#> [22] scales_1.1.0             genefilter_1.68.0        RBGL_1.62.1             
#> [25] askpass_1.1              rappdirs_0.3.1           stringr_1.4.0           
#> [28] digest_0.6.25            rmarkdown_2.1            AnnotationForge_1.28.0  
#> [31] jpeg_0.1-8.1             pkgconfig_2.0.3          htmltools_0.4.0         
#> [34] BSgenome_1.54.0          dbplyr_1.4.2             limma_3.42.2            
#> [37] highr_0.8                rlang_0.4.5              RSQLite_2.2.0           
#> [40] GOstats_2.52.0           hwriter_1.3.2            dplyr_0.8.4             
#> [43] VariantAnnotation_1.32.0 RCurl_1.98-1.1           magrittr_1.5            
#> [46] GO.db_3.10.0             GenomeInfoDbData_1.2.2   Matrix_1.2-18           
#> [49] Rcpp_1.0.3               munsell_0.5.0            lifecycle_0.1.0         
#> [52] stringi_1.4.6            yaml_2.2.1               edgeR_3.28.1            
#> [55] zlibbioc_1.32.0          BiocFileCache_1.10.2     grid_3.6.2              
#> [58] blob_1.2.1               crayon_1.3.4             lattice_0.20-40         
#> [61] splines_3.6.2            GenomicFeatures_1.38.2   annotate_1.64.0         
#> [64] hms_0.5.3                batchtools_0.9.12        locfit_1.5-9.1          
#> [67] knitr_1.28               pillar_1.4.3             rjson_0.2.20            
#> [70] base64url_1.4            biomaRt_2.42.0           XML_3.99-0.3            
#> [73] glue_1.3.1               evaluate_0.14            latticeExtra_0.6-29     
#> [76] data.table_1.12.8        png_0.1-7                vctrs_0.2.3             
#> [79] gtable_0.3.0             openssl_1.4.1            purrr_0.3.3             
#> [82] assertthat_0.2.1         ggplot2_3.3.0            xfun_0.12               
#> [85] xtable_1.8-4             survival_3.1-8           pheatmap_1.0.12         
#> [88] tibble_2.1.3             AnnotationDbi_1.48.0     memoise_1.1.0           
#> [91] brew_1.0-6               GSEABase_1.48.0

[dcassol]

Hi @FelixErnst, Thank you for replying to us. I modified the conditional for checking the modulecmd is available, and pushed to the master branch. Could you please install systemPipeR v1.21.4 from GitHub and try again?

if (!requireNamespace("BiocManager", quietly=TRUE))
    install.packages("BiocManager")
BiocManager::install("tgirke/systemPipeR", build_vignettes=TRUE, dependencies=TRUE)

I appreciate your time to try this.

[FelixErnst]

I tried it. the message is gone. However, two error are reported, which are:

args <- runCommandline(args = args_rtRNA, make_bam=TRUE)
#> [1] "Loading module NA"
#> ERROR: Unable to locate a modulefile for 'NA'
#> ERROR: Unable to locate a modulefile for 'bwa/0.7.17'

The second one I expected. I am not sure, where the NA comes from. The output is racey, since it only occurs in a fresh R session, after creating the args_rtRNA freshly using

args_rtRNA <- loadWorkflow(targets = targets, wf_file = "bwa-pe-rtRNA.cwl", 
                           input_file = "bwa-pe-rtRNA.yml", dir_path = cwl_path)
args_rtRNA <- renderWF(args_rtRNA, inputvars = c(FileName1 = "_FASTQ_PATH1_", FileName2 = "_FASTQ_PATH2_", 
                                                 SampleName = "_SampleName_"))

However, this happens also with the default example bwa-pe.cwl and .yml from the example vignette.

modulecmd is located correctly

system("which modulecmd",intern=TRUE,ignore.stderr=TRUE)
#> [1] "/usr/bin/modulecmd"

After removing environment-modules the default message is returning, which I did expect after your explanation. Thanks again for the clarification.

Regarding the whole thing, I have a bit of a suggestion. Since I tried systemPipeR as a bit of a naive user, I was totally confused by the message and the fact the command seemed to work. I noticed that the error message does occur again here https://github.com/tgirke/systemPipeR/blob/e09213184d76570d919986821bef9f0781a3bc8e/R/utilities.R#L218 if a command fails. Maybe you want to remove the message displayed now, if the environment-modules are not installed and only display it, if a command does not work. This might lead everybody down the right road.

[dcassol]

Hi @FelixErnst, Thank you again for trying and report the findings. My big guess here is you may not have the same version of BWA available on the module system. Also, this is the reason I set up the stop message you mentioned. Otherwise, system function used internally will return an error, which most of the cases don't make sense for the user. In this case, the tryCatch function prevents that, and returns a meaningful message, explaining you don't have the software set in the current PATH. We specified a version on the *.yml file (bwa/0.7.17), and maybe you don't have the same specific version or the name of the software is set up in a different away. You can always test as follow:

module("list") # It will list loaded software in the current session
module("avail") # It will list all available software from the module system, you can check for "bwa/0.7.17'"
modules(args_rtRNA)  # It will display what is the current software/version it will be loaded
module("load", modules(args_rtRNA)) #  Example for loading a software into the shell environment "manually", exactly what "runCommandline" does internally
module("load", "BWA")  # If I type the wrong name or format, I also will get the error
# >ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'BWA' 

I tried the param files you share with me, and I couldn't reproduce the same NA you got it. Please, could you make sure you have available bwa in your module environment, and let me know if you still get the NA error.

Thank you very much, Dani

[FelixErnst]

Hi @dcassol,

Thanks again for taking the time for the comprehensive answer. I will try to debug this on my end, but I guess it will take some time. I am not 100% familiar with the module system and have to read up on it.

I will close this issue for now.

Thanks again. Felix

[dcassol]

Hi @FelixErnst, Thank you for using systemPipeR and report your experience and the bugs! This helps us a lot to improve the package! Feel free to re-open if you have any additional questions. Cheers, Dani