zinbFit again stops with error

[friedue]

Alright, I realized I needed to re-run the zinbFit routine -- and I'm again running into the same error that I described before in the zingeRzinbwave repo

library(zinbwave)
set.seed(1133)
library(doParallel)
library(BiocParallel)

NCORES <- 6
registerDoParallel(NCORES)
register(DoparParam())

## loading all cells
##for james_DE2:
load("sce.filt.rda")
condition <- factor(colData(sce.filt)$condition, levels = c("W","H","D"), ordered = TRUE)
cts <- counts(sce.filt)
core <- SummarizedExperiment(cts, colData = data.frame(condition = condition))
design <- model.matrix(~condition)

## fit the ZINB regression model & compute ZINB-WaVE observational weights
zinb <- zinbFit(core, X = design, commondispersion = TRUE, epsilon=1e12, verbose = TRUE, maxiter.optimize = 10)

Create model:
ok
Initialize parameters:
Error in updt[[as.integer(i)]] <- .error(msg) : 
  attempt to select more than one element in integerOneIndex
In addition: Warning message:
In updt[[as.integer(i)]] <- .error(msg) : NAs introduced by coercion

session info:

attached base packages:
[1] parallel  stats4    stats     graphics  grDevices utils     datasets
[8] methods   base

other attached packages:
 [1] BiocParallel_1.12.0        doParallel_1.0.11
 [3] iterators_1.0.9            foreach_1.4.4
 [5] zinbwave_1.3.0             SingleCellExperiment_1.0.0
 [7] SummarizedExperiment_1.8.1 DelayedArray_0.4.1
 [9] matrixStats_0.53.1         Biobase_2.38.0
[11] GenomicRanges_1.30.1       GenomeInfoDb_1.14.0
[13] IRanges_2.12.0             S4Vectors_0.16.0
[15] BiocGenerics_0.24.0

loaded via a namespace (and not attached):
 [1] pcaPP_1.9-73           Rcpp_0.12.15           pillar_1.1.0
 [4] compiler_3.4.3         XVector_0.18.0         bitops_1.0-6
 [7] tools_3.4.3            zlibbioc_1.24.0        digest_0.6.15
[10] bit_1.1-12             memoise_1.1.0          tibble_1.4.2
[13] annotate_1.56.1        RSQLite_2.0            lattice_0.20-35
[16] pspline_1.0-18         rlang_0.1.6            Matrix_1.2-12
[19] DBI_0.7                mvtnorm_1.0-7          GenomeInfoDbData_1.0.0
[22] copula_0.999-18        genefilter_1.60.0      glmnet_2.0-16
[25] bit64_0.9-7            locfit_1.5-9.1         grid_3.4.3
[28] ADGofTest_0.3          AnnotationDbi_1.40.0   survival_2.41-3
[31] XML_3.98-1.9           limma_3.34.8           blob_1.1.0
[34] edgeR_3.20.8           codetools_0.2-15       splines_3.4.3
[37] stabledist_0.7-1       softImpute_1.4         xtable_1.8-2
[40] numDeriv_2016.8-1      gsl_1.9-10.3           RCurl_1.95-4.10

[drisso]

Can you please try the same code with register(SerialParam())?

I find that sometimes BiocParallel masks more meaningful error messages.

It seems that the problem is with zinbInitialize(). Can you try running only that step and see if you still get the error?

Also, it would be great to have a smaller, reproducible example to test things out.

[friedue]

I re-started the process without parallellization (core = 1) and it finished successfully (after trials with 6, 10 and more cores failed during the same session).

[drisso]

One possibility is that it's a memory issue, as more cores will require more memory. Does it always fail at the initialization step? That's probably not the most memory intensive step...

[friedue]

Does it always fail at the initialization step

Hm, can't say for sure. I tend towards: no. Definitely not when I was still working with the previous version, that usually failed towards the last iteration (making it even more infuriating ;) )

[drisso]

Hi @friedue ,

I just wanted to let you know that I have experienced a similar issue (zinbwave failing randomly at either the initialization step or at the very last iteration...) on a cluster -- perhaps even the same cluster? ;)

I think that that might be related to the doParallel package. Have you tried using:

library(BiocParallel)
register(bpstart(MulticoreParam(workers=NCORES)))

Instead of

registerDoParallel(NCORES)
register(DoparParam())

I believe this is a problem with BiocParallel but I still have to pin it down exactly...

[friedue]

not the same cluster, but a server maintained by the same people :)

I have not tried the BiocParallel approach, but I will probably use zinbwave some time soon, so if you find that to work more reliably, let me know!