Open AharoniLab opened 1 year ago
Hi, could you link any other issue you refer to ? Could you give a code snippet that triggered the issue ?
Can you run traceback()
after the error happened ? Yours, Steffen
Referring to:
https://github.com/sneumann/xcms/issues/437
The following command already triggers this error. 'rd_pools' in this case is an "OnDiskMSnExp" object linked to four experimental files:
xd_pools = findChromPeaks (rd_pools,cwp,BPPARAM = bpparam$MulticoreParam)
| | 0%Error in result[[njob]] <- value :
attempt to select less than one element in OneIndex
In addition: Warning message:
In parallel::mccollect(wait = FALSE, timeout = 1) :
1 parallel job did not deliver a result
traceback()
7: bploop.lapply(cls, X, lapply, ARGFUN, BPPARAM)
6: bploop(cls, X, lapply, ARGFUN, BPPARAM)
5: bplapply(object_mslevel, FUN = findChromPeaks_OnDiskMSnExp, method = "centWave",
param = param, BPPARAM = BPPARAM)
4: bplapply(object_mslevel, FUN = findChromPeaks_OnDiskMSnExp, method = "centWave",
param = param, BPPARAM = BPPARAM)
3: .local(object, param, ...)
2: findChromPeaks(rd_pools, cwp, BPPARAM = bpparam$MulticoreParam)
1: findChromPeaks(rd_pools, cwp, BPPARAM = bpparam$MulticoreParam)`
Thanks, Nir.
Thanks Nir for the additional information. Unfortunately I do not have any direct suggestion.
There is https://bioconductor.org/packages/release/bioc/vignettes/BiocParallel/inst/doc/Errors_Logs_And_Debugging.pdf
and things to try could be adding sink(NULL, type = "message")
(see 2.1 Messages and warnings) into
https://github.com/sneumann/xcms/blob/af1e8b9e79b6039a34951e9e79329d99ce778397/R/methods-OnDiskMSnExp.R#L83
and/or getting the traceback via the instructions in "2.3 Identify failures with bpok()"
using the equivalent of result <- bptry(bplapply(list(1, "2", 3), sqrt, BPPARAM=param))
and
tail(attr(result[[which(!bpok(result))]], "traceback
.
If any of that gives (more) informative feedback, that'd be really great. Any idea how I could provoke the error here on my machine ?
Yours,
Steffen
Hm, maybe there is a simpler way to get the debug info:
bpparam <- MulticoreParam(workers = 4,
stop.on.error = TRUE,
log = TRUE, threshold = "INFO", logdir = "/tmp")
but I got empty log files in /tmp
, maybe because I was unable to trigger the issue.
Yours, Steffen
Hi Nir,
since you don't get the error with SerialParam
but only with MulticoreParam
I suspect it might have to do with the cluster settings. On our HPC we use slurm as queueing system and usually all works well, only thing is I need to make sure that all parallel processes are run on the same node. MulticoreParam
uses shared memory, thus it will not work if the processes are run on different nodes (servers/computers). Also, make sure you assign enough memory to each node/task - and obviously also each node (process) needs to have access to the raw data file (same path). Maybe you could also try to use SnowParam
as an alternative?
Hi,
Thanks both for your replies.
1) running with SnowParam
did not work either and resulted in the following error:
Error in serialize(data, node$con) : ignoring SIGPIPE signal
Calls: local ... .send -> MulticoreParam
did not result in any log files on this server, but did generate the per process logs with the run which completely successfully on the non-cluster PC.
3) generating logs using the suggested bptry
on bpapply
did not work either as findChromPeaks
kept throwing errors regarding the split object type:
rd_pools_byFile = split (rd_pools,f=fromFile (rd_pools))
result <- bptry (
bplapply (rd_pools_byFile,findChromPeaks,BPPARAM=bpparam)
)
Error in (function (classes, fdef, mtable) : unable to find an inherited method for function ‘findChromPeaks’ for signature ‘"OnDiskMSnExp", "missing"’
OnDiskMSnExp
object?!Can you please provide the output of sessionInfo()
from the cluster as well the local (working) system?
I got also a bit confused by the findChromPeaks(rd_pools, cwp, BPPARAM = bpparam$MulticoreParam)
call - ideally you should run it either with
register(bpstart(MulticoreParam(4)))
res <- findChromPeaks(rd_pools, cwp)
or
res <- findChromPeaks(rd_pools, cwp, BPPARAM = MulticoreParam(4))
The first option should be preferred as you define and initialize the parallel processing setup globally for your R session. Any xcms
function supporting parallel processing will automatically use this default setup.
Regarding the error you got: the problem is that you forgot to pass the cwp
parameter. That's why the error says unable to find an inherited... for signature "OnDiskMSnExp", "missing". You should rather call
rd_pools_byFile = split (rd_pools,f=fromFile (rd_pools))
result <- bptry (
bplapply (rd_pools_byFile,findChromPeaks, param = cwp, BPPARAM=bpparam)
)
Having my own parallel processing problems on a linux system in a conda environment. Using an older version of R/XCMS/BioC works fine on the same computer. I am pretty ignorant when it comes to HPC, so maybe i am failing to set something properly, but the fact that it works with older R suggests that something about BioC paralell processing has changed.
proj.dir <- "20220310-CBROE-CB-1173/PH_QC_Subset/" library(xcms)
register(bpstart(MulticoreParam(workers = 12))) setwd(paste0("/home/pmflab/RStor/Projects/", proj.dir)) seq <- read.csv('seq_QCs.csv', header = TRUE, check.names = FALSE) # seq <- seq[1:2,] mzml <- c(paste0(as.character(seq[,1]), ".mzmL")) ## 28 files for testing
pd <- data.frame("filename" = mzml, seq[,2]) qc <- grepl("QC", seq[,2], ignore.case = TRUE) sample.groups <- as.integer(qc)
cw.param <- CentWaveParam(
ppm = 25,
peakwidth = c(2.2, 15),
snthresh = 10,
prefilter = c(3, 10),
mzCenterFun = "wMean",
integrate = 1L,
mzdiff = 0.015,
fitgauss = TRUE,
noise = 2,
verboseColumns = TRUE,
roiList = list(),
firstBaselineCheck = TRUE,
roiScales = numeric(),
extendLengthMSW = TRUE
)
raw_data <- readMSData(files = mzml, pdata = new("NAnnotatedDataFrame", pd), mode = "onDisk") x.data <- findChromPeaks(raw_data, param = cw.param)
Gives an error: Error in serialize(data, node$con, xdr = FALSE) : ignoring SIGPIPE signal Error in serialize(data, node$con, xdr = FALSE) : error writing to connection
this is operating in a Conda environment, R 4.2.2.
sessionInfo()
R version 4.2.2 (2022-10-31)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: Ubuntu 20.04.4 LTS
Matrix products: default BLAS/LAPACK: /home/pmflab/miniconda3/envs/r422/lib/libopenblasp-r0.3.21.so
locale: [1] C
attached base packages: [1] stats4 stats graphics grDevices utils datasets methods [8] base
other attached packages: [1] xcms_3.20.0 MSnbase_2.24.0 ProtGenerics_1.30.0 [4] S4Vectors_0.36.0 mzR_2.32.0 Rcpp_1.0.9 [7] Biobase_2.58.0 BiocGenerics_0.44.0 BiocParallel_1.32.0
loaded via a namespace (and not attached): [1] lattice_0.20-45 assertthat_0.2.1 [3] digest_0.6.30 foreach_1.5.2 [5] utf8_1.2.2 R6_2.5.1 [7] GenomeInfoDb_1.34.1 plyr_1.8.8 [9] mzID_1.36.0 ggplot2_3.4.0 [11] pillar_1.8.1 zlibbioc_1.44.0 [13] rlang_1.0.6 Matrix_1.5-3 [15] preprocessCore_1.60.0 RCurl_1.98-1.9 [17] munsell_0.5.0 DelayedArray_0.24.0 [19] compiler_4.2.2 MsFeatures_1.6.0 [21] pkgconfig_2.0.3 pcaMethods_1.90.0 [23] tidyselect_1.2.0 SummarizedExperiment_1.28.0 [25] tibble_3.1.8 GenomeInfoDbData_1.2.9 [27] RANN_2.6.1 IRanges_2.32.0 [29] codetools_0.2-18 matrixStats_0.62.0 [31] XML_3.99-0.12 fansi_1.0.3 [33] dplyr_1.0.10 MASS_7.3-58.1 [35] bitops_1.0-7 MassSpecWavelet_1.64.0 [37] grid_4.2.2 gtable_0.3.1 [39] lifecycle_1.0.3 affy_1.76.0 [41] DBI_1.1.3 magrittr_2.0.3 [43] MsCoreUtils_1.10.0 scales_1.2.1 [45] ncdf4_1.19 cli_3.4.1 [47] impute_1.72.0 XVector_0.38.0 [49] affyio_1.68.0 doParallel_1.0.17 [51] limma_3.54.0 robustbase_0.95-0 [53] generics_0.1.3 vctrs_0.5.1 [55] RColorBrewer_1.1-3 iterators_1.0.14 [57] tools_4.2.2 glue_1.6.2 [59] DEoptimR_1.0-11 MatrixGenerics_1.10.0 [61] parallel_4.2.2 clue_0.3-62 [63] colorspace_2.0-3 cluster_2.1.4 [65] BiocManager_1.30.19 vsn_3.66.0 [67] GenomicRanges_1.50.0 MALDIquant_1.22
This conda set up worked on the same computer: sessionInfo() R version 4.1.2 (2021-11-01) Platform: x86_64-conda-linux-gnu (64-bit) Running under: Ubuntu 20.04.4 LTS
Matrix products: default BLAS/LAPACK: /home/pmflab/miniconda3/envs/xcms/lib/libopenblasp-r0.3.18.so
locale: [1] C
attached base packages: [1] stats4 stats graphics grDevices utils datasets methods [8] base
other attached packages: [1] xcms_3.16.1 MSnbase_2.20.4 ProtGenerics_1.26.0 [4] S4Vectors_0.32.3 mzR_2.28.0 Rcpp_1.0.8 [7] Biobase_2.54.0 BiocGenerics_0.40.0 BiocParallel_1.28.3
loaded via a namespace (and not attached): [1] lattice_0.20-45 assertthat_0.2.1 [3] digest_0.6.29 foreach_1.5.2 [5] utf8_1.2.2 R6_2.5.1 [7] GenomeInfoDb_1.30.1 plyr_1.8.6 [9] mzID_1.32.0 ggplot2_3.3.5 [11] pillar_1.7.0 zlibbioc_1.40.0 [13] rlang_1.0.1 Matrix_1.4-0 [15] preprocessCore_1.56.0 RCurl_1.98-1.6 [17] munsell_0.5.0 DelayedArray_0.20.0 [19] compiler_4.1.2 MsFeatures_1.2.0 [21] pkgconfig_2.0.3 pcaMethods_1.86.0 [23] tidyselect_1.1.1 SummarizedExperiment_1.24.0 [25] GenomeInfoDbData_1.2.7 tibble_3.1.6 [27] RANN_2.6.1 IRanges_2.28.0 [29] codetools_0.2-18 matrixStats_0.61.0 [31] XML_3.99-0.8 fansi_1.0.2 [33] crayon_1.4.2 dplyr_1.0.8 [35] bitops_1.0-7 MASS_7.3-55 [37] MassSpecWavelet_1.60.0 grid_4.1.2 [39] gtable_0.3.0 lifecycle_1.0.1 [41] affy_1.72.0 DBI_1.1.2 [43] magrittr_2.0.2 MsCoreUtils_1.6.0 [45] scales_1.1.1 ncdf4_1.19 [47] cli_3.1.1 impute_1.68.0 [49] XVector_0.34.0 affyio_1.64.0 [51] doParallel_1.0.17 limma_3.50.0 [53] robustbase_0.93-9 ellipsis_0.3.2 [55] generics_0.1.2 vctrs_0.3.8 [57] RColorBrewer_1.1-2 iterators_1.0.14 [59] tools_4.1.2 glue_1.6.1 [61] DEoptimR_1.0-10 purrr_0.3.4 [63] MatrixGenerics_1.6.0 parallel_4.1.2 [65] clue_0.3-60 colorspace_2.0-2 [67] cluster_2.1.2 BiocManager_1.30.16 [69] vsn_3.62.0 GenomicRanges_1.46.1 [71] MALDIquant_1.21
i was able to get this to work in Linux in new xcms/Bioc by moving to Snow:
Success: register(bpstart(SnowParam(workers = 12))) Fails: register(bpstart(MulticoreParam(workers = 12)))
sessionInfo() R version 4.2.2 (2022-10-31) Platform: x86_64-conda-linux-gnu (64-bit) Running under: Ubuntu 20.04.4 LTS
Matrix products: default BLAS/LAPACK: /home/pmflab/miniconda3/envs/r422/lib/libopenblasp-r0.3.21.so
locale: [1] C
attached base packages: [1] stats4 stats graphics grDevices utils datasets methods [8] base
other attached packages: [1] xcms_3.20.0 MSnbase_2.24.0 ProtGenerics_1.30.0 [4] S4Vectors_0.36.0 mzR_2.32.0 Rcpp_1.0.9 [7] Biobase_2.58.0 BiocGenerics_0.44.0 BiocParallel_1.32.0
loaded via a namespace (and not attached): [1] lattice_0.20-45 assertthat_0.2.1 [3] digest_0.6.30 foreach_1.5.2 [5] utf8_1.2.2 R6_2.5.1 [7] GenomeInfoDb_1.34.1 plyr_1.8.8 [9] mzID_1.36.0 ggplot2_3.4.0 [11] pillar_1.8.1 zlibbioc_1.44.0 [13] rlang_1.0.6 Matrix_1.5-3 [15] preprocessCore_1.60.0 RCurl_1.98-1.9 [17] munsell_0.5.0 DelayedArray_0.24.0 [19] compiler_4.2.2 MsFeatures_1.6.0 [21] pkgconfig_2.0.3 pcaMethods_1.90.0 [23] tidyselect_1.2.0 SummarizedExperiment_1.28.0 [25] tibble_3.1.8 GenomeInfoDbData_1.2.9 [27] RANN_2.6.1 IRanges_2.32.0 [29] codetools_0.2-18 matrixStats_0.62.0 [31] XML_3.99-0.12 fansi_1.0.3 [33] dplyr_1.0.10 MASS_7.3-58.1 [35] bitops_1.0-7 MassSpecWavelet_1.64.0 [37] grid_4.2.2 gtable_0.3.1 [39] lifecycle_1.0.3 affy_1.76.0 [41] DBI_1.1.3 magrittr_2.0.3 [43] MsCoreUtils_1.10.0 scales_1.2.1 [45] ncdf4_1.19 cli_3.4.1 [47] impute_1.72.0 XVector_0.38.0 [49] affyio_1.68.0 doParallel_1.0.17 [51] limma_3.54.0 robustbase_0.95-0 [53] generics_0.1.3 vctrs_0.5.1 [55] RColorBrewer_1.1-3 iterators_1.0.14 [57] tools_4.2.2 glue_1.6.2 [59] DEoptimR_1.0-11 MatrixGenerics_1.10.0 [61] parallel_4.2.2 clue_0.3-62 [63] colorspace_2.0-3 cluster_2.1.4 [65] BiocManager_1.30.19 vsn_3.66.0 [67] GenomicRanges_1.50.0 MALDIquant_1.22
Hi back,
Sorry for the pause. Following the info which Steffen sent, the sys. admin has limited this instance's use of resources to 80% and this has already made the command work as-is also when using multi cores. Ee also set the VM "overcommit_ratio" to zero - but I cannot say if this directly resolved the issue without the extra limits on the VM.
@jorainer
_register(bpstart(MulticoreParam(4))) res <- findChromPeaks(rdpools, cwp)
I initially used this setup, but I have also had occasional problems when running 'chromatogram' and 'fillChromPeaks' with the multi-core option - so I resolved to running different XCMS functions with different multi/series-core settings. Still 'bptry' on the split data might be a better way to deal with very variable system loads and possible crashes in one of the threads.
Regarding the error you got: the problem is that you forgot to pass the cwp parameter. That's why the error says unable to find an inherited... for signature "OnDiskMSnExp", "missing
I could not find any missing 'cwp' parameter - but regardless you must admit that the error message does not contribute much to debugging the problem...
Cluster VM info: `> sessionInfo() R version 3.6.3 (2020-02-29) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Debian GNU/Linux bullseye/sid
Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0 LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0
locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages: [1] stats graphics grDevices utils datasets methods base `
Local PC info: `> sessionInfo() R version 3.6.3 (2020-02-29) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 18.04.6 LTS
Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3 LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so
locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages: [1] grid stats4 parallel stats graphics grDevices utils [8] datasets methods base
other attached packages: [1] Rdisop_1.46.0 classyfireR_0.3.6 magrittr_2.0.1 [4] gridExtra_2.3 readxl_1.3.1 RColorBrewer_1.1-2 [7] XML_3.99-0.3 CAMERA_1.42.0 xcms_3.8.2 [10] MSnbase_2.12.0 ProtGenerics_1.18.0 S4Vectors_0.24.4 [13] mzR_2.20.0 Rcpp_1.0.6 BiocParallel_1.20.1 [16] Biobase_2.46.0 BiocGenerics_0.32.0
loaded via a namespace (and not attached): [1] vsn_3.54.0 splines_3.6.3 foreach_1.5.1 [4] Formula_1.2-4 BiocManager_1.30.18 affy_1.64.0 [7] latticeExtra_0.6-29 RBGL_1.62.1 cellranger_1.1.0 [10] robustbase_0.93-7 impute_1.60.0 pillar_1.4.7 [13] backports_1.2.1 lattice_0.20-45 glue_1.4.2 [16] limma_3.42.2 digest_0.6.27 checkmate_2.0.0 [19] colorspace_2.0-0 htmltools_0.5.1.1 preprocessCore_1.48.0 [22] Matrix_1.3-4 plyr_1.8.6 MALDIquant_1.19.3 [25] pkgconfig_2.0.3 zlibbioc_1.32.0 purrr_0.3.4 [28] scales_1.1.1 RANN_2.6.1 jpeg_0.1-8.1 [31] affyio_1.56.0 tibble_3.0.6 htmlTable_2.1.0 [34] generics_0.1.0 IRanges_2.20.2 ggplot2_3.3.3 [37] ellipsis_0.3.1 nnet_7.3-16 MassSpecWavelet_1.52.0 [40] survival_3.2-13 crayon_1.4.1 ncdf4_1.17 [43] doParallel_1.0.16 MASS_7.3-54 foreign_0.8-76 [46] graph_1.64.0 data.table_1.13.6 tools_3.6.3 [49] lifecycle_1.0.0 stringr_1.4.0 munsell_0.5.0 [52] cluster_2.1.1 pcaMethods_1.78.0 compiler_3.6.3 [55] mzID_1.24.0 rlang_0.4.10 iterators_1.0.13 [58] rstudioapi_0.13 htmlwidgets_1.5.3 igraph_1.2.6 [61] base64enc_0.1-3 gtable_0.3.0 codetools_0.2-18 [64] multtest_2.42.0 R6_2.5.0 knitr_1.31 [67] dplyr_1.0.4 Hmisc_4.4-2 stringi_1.5.3 [70] vctrs_0.3.6 rpart_4.1-15 png_0.1-7 [73] DEoptimR_1.0-8 tidyselect_1.1.0 xfun_0.21`
I could not find any missing 'cwp' parameter - but regardless you must admit that the error message does not contribute much to debugging the problem...
sorry, I was maybe not clear. What I meant is the following:
the code you used and that has thrown this error message (for signature "OnDiskMSnExp", "missing") was:
rd_pools_byFile = split (rd_pools,f=fromFile (rd_pools))
result <- bptry (
bplapply (rd_pools_byFile,findChromPeaks,BPPARAM=bpparam)
)
so, in fact you are applying findChromPeaks
to each element of rd_pools_byFile
, but you are not passing the parameter object to the findChromPeaks
function. So, in essence, in each loop you are calling findChromPeaks(rd_pools_byFile[[i]])
and findChromPeaks
cries that it can not find the parameter object (param
is in fact "missing"
, that's what the cryptic error message means).
So, you would need to pass also the cwp
with parameter param
to findChromPeaks
:
rd_pools_byFile = split (rd_pools,f=fromFile (rd_pools))
result <- bptry (
bplapply (rd_pools_byFile, findChromPeaks, param = cwp, BPPARAM=bpparam)
)
hope this makes sense
Hi,
I am getting the same cryptic error mentioned a couple of years ago
With the following added details:
Any update or feedback would be welcome.
Thanks!