CreateSinglerObject on hpc (caught segfault)

[hjc0818]

Hi,

Previously I was using SingleR on my own laptop, which can handle around 5000 cells with fine tuning. However I want to analyze 50000 cells data so I installed Single R on HPC running with R/3.5.0. After running for 3 to 4 hours it always give me the error:

my output:

[1] "Dimensions of counts data: 16127x52693"
[1] "Annotating data with Immgen..."
[1] "Variable genes method: de"
[1] "Number of DE genes:3121"
[1] "Number of cells: 52693"
[1] "Fine-tuning round on top cell types (using 8 CPU cores):"
[1] "Number of DE genes:3121"
[1] "Number of clusters: 10"
[1] "Fine-tuning round on top cell types (using 8 CPU cores):"
[1] "Annotating data with Immgen (Main types)..."
[1] "Number of DE genes:2013"
[1] "Number of cells: 52693"
[1] "Fine-tuning round on top cell types (using 8 CPU cores):"
~                                                              

System output file:

------------------------------------------------------------

Successfully completed.

Resource usage summary:

    CPU time :                                   70723.40 sec.
    Max Memory :                                 104 GB
    Average Memory :                             57.05 GB
    Total Requested Memory :                     180.00 GB
    Delta Memory :                               76.00 GB
    Max Swap :                                   -
    Max Processes :                              12
    Max Threads :                                13
    Run time :                                   15481 sec.
    Turnaround time :                            16104 sec.
The output (if any) follows:

During startup - Warning message:
Setting LC_CTYPE failed, using "C"
Loading required package: ggplot2
Loading required package: cowplot

Attaching package: 'cowplot'

The following object is masked from 'package:ggplot2':

    ggsave

Loading required package: Matrix
Warning message:
replacing previous import 'BiocGenerics::dims' by 'Biobase::dims' when loading 'AnnotationDbi'

 *** caught segfault ***
address 0x2aadc852ee90, cause 'memory not mapped'

Traceback:
 1: hclust(dist(scores, method = "euclidean"), method = "ward.D2")
 2: SingleR.Cluster(singler$SingleR.single.main, 10)
 3: SingleR.CreateObject(sc.data.gl, x, clusters, species, citation,     technology, do.main.types = do.main.types, variable.genes = variable.genes,     fine.tune = fine.tune, numCores = numCores)
 4: FUN(X[[i]], ...)
 5: lapply(ref.list, FUN = function(x) {    SingleR.CreateObject(sc.data.gl, x, clusters, species, citation,         technology, do.main.types = do.main.types, variable.genes = variable.genes,         fine.tune = fine.tune, numCores = numCores)})
 6: CreateSinglerObject(counts = treated.cca@data, annot = NULL,     project.name = "treated", technology = "10X", species = "Mouse",     citation = "", ref.list = list(immgen), normalize.gene.length = F,     variable.genes = "de", fine.tune = T, do.signatures = T,     clusters = NULL, do.main.types = T, reduce.file.size = T,     numCores = 8)
An irrecoverable exception occurred. R is aborting now ...
/home/jhan6/.lsbatch/1549232611.536704.shell: line 26: 29140 Segmentation fault      (core dumped) Rscript run_singler.R > run_singler.out
My R jobs are done.
Sun Feb  3 20:51:55 CST 2019
My job ran on the following host:
cdragon096

How can I solve this problem?

Sincere

Han.

[dviraran]

Hi,

I am not sure why it failed at this point. It seems like it was able to perform the fine-tuning step using Immgen, which is the most exhausting process, and failed while fine-tuning with main cell types, which is much less exhausting. Anyhow, for big datasets you can try breaking it to smaller pieces - see this tutorial for more details.

Best, Dvir

[hjc0818]

Thank you Dvir, I will try this and also I will try setting main types to F. If I solve the problem I will feedback again.

Sincere

Han.