stop worker failed, Error in serialize, error writing to connection

[etrh]

I have used IPO before, but this is the first time I am seeing this error.

I have 26 files in data and the machine I am working on has 160 processors.

## FIRST TRY

Error: stop worker failed: 'clear_cluster' receive data failed: reached elapsed time limit In addition: Warning message: In optimizeXcmsSet(files = data, params = peakpickingParameters, : IPO (nSlaves-argument) and xcms (BPPARAM-argument) parallelisation cannot be used together! Setting nSlaves to 1 Timing stopped at: 1342 589.8 400.8

Error in serialize(data, node$con, xdr = FALSE) : error writing to connection In addition: Warning message: In optimizeXcmsSet(files = data, params = peakpickingParameters, : IPO (nSlaves-argument) and xcms (BPPARAM-argument) parallelisation cannot be used together! Setting nSlaves to 1

The script I am trying to run (which leads to the above error) is as follows

peakpickingParameters <- IPO::getDefaultXcmsSetStartingParams('centWave')
peakpickingParameters$min_peakwidth <- c(12,45)
peakpickingParameters$max_peakwidth <- c(15, 75)
peakpickingParameters$ppm <- c(5,30)
peakpickingParameters$snthresh <- c(3,100)
peakpickingParameters$noise <- c(2000,8000)
peakpickingParameters$prefilter <- c(3,10)
peakpickingParameters$value_of_prefilter <- c(2000,20000)
peakpickingParameters$fitgauss <- TRUE
pcores <- 100

optimized.xcmsSet <- system.time({
    resultPeakpicking <- optimizeXcmsSet(files = data,
                                         params = peakpickingParameters,
                                         subdir = "IPO_plots", nSlaves = pcores)
})

##SECOND TRY

If I set nSlaves = 1 (inside the optimizeXcmsSet function), then I get this error:

Error: stop worker failed: 'clear_cluster' receive data failed: reached elapsed time limit Timing stopped at: 1402 363 459.3

Error in serialize(data, node$con, xdr = FALSE) : error writing to connection

##THIRD TRY

And then if I do the following just before running IPO (and without specifying the nSlaves parameter at all, neither inside optimizeXcmsSet nor as peakpickingParameters$nSlaves):

library(BiocParallel)
register(MulticoreParam(100))

I get this error:

Error in xcms::xcmsSet(files = files, method = "centWave", peakwidth = c(xcmsSetParameters$min_peakwidth[task], : Chromatographic peak detection failed for all files! The first error was: Error in seq.default(from = scalerange[1], to = scalerange[2], by = 2): wrong sign in 'by' argument In addition: Warning message: In optimizeXcmsSet(files = data, params = peakpickingParameters, : IPO (nSlaves-argument) and xcms (BPPARAM-argument) parallelisation cannot be used together! Setting nSlaves to 1 Timing stopped at: 1446 345.3 446.1

##FOURTH TRY

Then I tried something different. I introduced peakpickingParameters$nSlaves <- 6 and then ran IPO with nSlaves = 5 and using only two data files.

optimized.xcmsSet <- system.time({ resultPeakpicking <- optimizeXcmsSet(files = data[c(3,21)], params = peakpickingParameters,nSlaves = 5, subdir = "IPO_plots") })

And this gave me an error again:

Error in xcms::xcmsSet(files = files, method = "centWave", peakwidth = c(xcmsSetParameters$min_peakwidth[task], : Chromatographic peak detection failed for all files! The first error was: Error in seq.default(from = scalerange[1], to = scalerange[2], by = 2): wrong sign in 'by' argument In addition: Warning messages: 1: In optimizeXcmsSet(files = data[c(3, 21)], params = peakpickingParameters, : IPO (nSlaves-argument) and xcms (BPPARAM-argument) parallelisation cannot be used together! Setting nSlaves to 1 2: In optimizeXcmsSet(files = data[c(3, 21)], params = peakpickingParameters, : Use of xcmsSet-argument 'nSlaves' is deprecated! Please use 'BPPARAM' instead. Timing stopped at: 70.14 4.524 124.1

##FIFTH TRY

Then I tried setting nSlaves = MulticoreParam(5) inside optimizeXcmsSet with two data files and peakpickingParameters$nSlaves <- 6 and I was still receiving an error:

Error in xcms::xcmsSet(files = files, method = "centWave", peakwidth = c(xcmsSetParameters$min_peakwidth[task], : Chromatographic peak detection failed for all files! The first error was: Error in seq.default(from = scalerange[1], to = scalerange[2], by = 2): wrong sign in 'by' argument In addition: Warning messages: 1: In optimizeXcmsSet(files = data[c(3, 21)], params = peakpickingParameters, : 'nSlaves' must be numeric! Setting it to 1. 2: In optimizeXcmsSet(files = data[c(3, 21)], params = peakpickingParameters, : Use of xcmsSet-argument 'nSlaves' is deprecated! Please use 'BPPARAM' instead. Timing stopped at: 75.18 7.335 131.8

What is exactly wrong here? My parameters? Or is the problem with the new version of xcms and R which are incompatible with IPO? @sneumann and @jotsetung, any idea?

R version 3.5.0 (2018-04-23)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: Scientific Linux 7.5 (Nitrogen)
> packageVersion("xcms")
[1] ‘3.0.2’
> packageVersion("IPO")
[1] ‘1.4.1’

[jorainer]

I used to get such time out errors on macOS and was able to solve that with pre-registering the parallel setup at the beginning in a way similar to your register(MulticoreParam(100)) call.

What I find interesting is your error message Error in seq.default(from = scalerange[1], to = scalerange[2], by = 2): wrong sign in 'by' argument. You would get such an error if scalerange is negative, which does not make sense. That variable should be positive. Have you tried to run the peak detection directly with xcms, not within IPO? Would be nice to track that error down.