search

RGLab / MAST

Tools and methods for analysis of single cell assay data in R
224 stars 57 forks source link

MAST uses all cores even after setting mc.cores #152

Closed fbnrst closed 3 years ago

fbnrst commented 3 years ago

Thanks for creating MAST!

For me, MAST always uses all cores when running zlm. I did set "options("mc.cores"=4)" and I also specified 

zlm(..., parallel = FALSE)

Still, when I run zlm all cores of my machine are used. Is there another option I do have to set?

R version 4.0.3 (2020-10-10)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: Debian GNU/Linux 10 (buster)

Matrix products: default
BLAS/LAPACK: /opt/conda/lib/libmkl_gf_lp64.so

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
 [1] parallel  stats4    tools     stats     graphics  grDevices utils    
 [8] datasets  methods   base     

other attached packages:
 [1] MAST_1.16.0                 SingleCellExperiment_1.12.0
 [3] SummarizedExperiment_1.20.0 Biobase_2.50.0             
 [5] GenomicRanges_1.42.0        GenomeInfoDb_1.26.0        
 [7] IRanges_2.24.0              S4Vectors_0.28.0           
 [9] BiocGenerics_0.36.0         MatrixGenerics_1.2.0       
[11] matrixStats_0.58.0         

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.6             plyr_1.8.6             compiler_4.0.3        
 [4] pillar_1.5.1           XVector_0.30.0         bitops_1.0-6          
 [7] zlibbioc_1.36.0        lifecycle_1.0.0        tibble_3.1.0          
[10] gtable_0.3.0           lattice_0.20-41        pkgconfig_2.0.3       
[13] rlang_0.4.10           Matrix_1.3-2           DBI_1.1.1             
[16] DelayedArray_0.16.0    GenomeInfoDbData_1.2.4 dplyr_1.0.5           
[19] stringr_1.4.0          generics_0.1.0         vctrs_0.3.6           
[22] tidyselect_1.1.0       grid_4.0.3             glue_1.4.2            
[25] data.table_1.14.0      R6_2.5.0               fansi_0.4.2           
[28] reshape2_1.4.4         purrr_0.3.4            ggplot2_3.3.3         
[31] magrittr_2.0.1         scales_1.1.1           ellipsis_0.3.1        
[34] assertthat_0.2.1       abind_1.4-5            colorspace_2.0-0      
[37] utf8_1.1.4             stringi_1.5.3          RCurl_1.98-1.2        
[40] munsell_0.5.0          crayon_1.4.1
amcdavid commented 3 years ago

So I understand the issue, you want only 4 cores to be used, but instead how many appear to be used? Can you provide the exact set of commands you are running.

On Mar 24, 2021, at 3:49 PM, Fabian Rost @.***> wrote:

Thanks for creating MAST!

For me, MAST always uses all cores when running zlm. I did set "options("mc.cores"=4)" and I also specified

zlm(..., parallel = FALSE) Still, when I run zlm all cores of my machine are used. Is there another option I do have to set?

R version 4.0.3 (2020-10-10) Platform: x86_64-conda-linux-gnu (64-bit) Running under: Debian GNU/Linux 10 (buster)

Matrix products: default BLAS/LAPACK: /opt/conda/lib/libmkl_gf_lp64.so

locale: [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] parallel stats4 tools stats graphics grDevices utils
[8] datasets methods base

other attached packages: [1] MAST_1.16.0 SingleCellExperiment_1.12.0 [3] SummarizedExperiment_1.20.0 Biobase_2.50.0
[5] GenomicRanges_1.42.0 GenomeInfoDb_1.26.0
[7] IRanges_2.24.0 S4Vectors_0.28.0
[9] BiocGenerics_0.36.0 MatrixGenerics_1.2.0
[11] matrixStats_0.58.0

loaded via a namespace (and not attached): [1] Rcpp_1.0.6 plyr_1.8.6 compiler_4.0.3
[4] pillar_1.5.1 XVector_0.30.0 bitops_1.0-6
[7] zlibbioc_1.36.0 lifecycle_1.0.0 tibble_3.1.0
[10] gtable_0.3.0 lattice_0.20-41 pkgconfig_2.0.3
[13] rlang_0.4.10 Matrix_1.3-2 DBI_1.1.1
[16] DelayedArray_0.16.0 GenomeInfoDbData_1.2.4 dplyr_1.0.5
[19] stringr_1.4.0 generics_0.1.0 vctrs_0.3.6
[22] tidyselect_1.1.0 grid_4.0.3 glue_1.4.2
[25] data.table_1.14.0 R6_2.5.0 fansi_0.4.2
[28] reshape2_1.4.4 purrr_0.3.4 ggplot2_3.3.3
[31] magrittr_2.0.1 scales_1.1.1 ellipsis_0.3.1
[34] assertthat_0.2.1 abind_1.4-5 colorspace_2.0-0
[37] utf8_1.1.4 stringi_1.5.3 RCurl_1.98-1.2
[40] munsell_0.5.0 crayon_1.4.1
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/RGLab/MAST/issues/152, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALLAHXFJDIJ2MYM3GCBBDLTFI657ANCNFSM4ZX6YMLQ.

fbnrst commented 3 years ago

exactly, this is what I wanted. Now, I tried to reproduce the issue, but I did some updates in between and I think it is gone. Thanks!

rdacemel commented 1 year ago

I have this very same problem, with MAST 1.26.0. I'm on a shared machine and it is literally taking up all. I guess some detectCores might be hidden/hardcoded somewhere? Even if I specify parallel = FALSE it still uses all it finds. 

options(mc.cores = 4)
zlmCond <- zlm( ~ broad_cell_type + cngeneson, sca, parallel = TRUE)
saveRDS(zlmCond, "MAST_zlm.RDS")
amcdavid commented 1 year ago

Please send sessionInfo() and an example that can be run from a fresh R session.

On Fri, Jun 16, 2023 at 1:32 AM rdacemel @.***> wrote:

I have this very same problem, with MAST 1.26.0. I'm on a shared machine and it is literally taking up all. I guess some detectCores might be hidden/hardcoded somewhere? Even if I specify parallel = FALSE it still uses all it finds.

options(mc.cores = 4) zlmCond <- zlm( ~ broad_cell_type + cngeneson, sca, parallel = FALSE) writeRDS(zlmCond, "MAST_zlm.RDS", parallel = TRUE)

— Reply to this email directly, view it on GitHub https://github.com/RGLab/MAST/issues/152#issuecomment-1594317759, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALLAHXEKLTAXJEWNERSCBDXLQK27ANCNFSM4ZX6YMLQ . You are receiving this because you commented.Message ID: @.***>

rdacemel commented 1 year ago

I think I figured something out. The multicore is working fine, but something underneath is trying to multithread as much as it can. I "fixed" it with: 

library(RhpcBLASctl)
blas_set_num_threads(4)

Then, if I parallel with 4 it uses 4 x 4 = 16 threads in total. I don't know exactly what this blas is doing or which package is calling it...