search

broadinstitute / infercnv

Inferring CNV from Single-Cell RNA-Seq
Other
556 stars 164 forks source link

node stack overflow #196

Closed QqQss closed 2 years ago

QqQss commented 4 years ago

Hey,

I trid to run the program by following commands:

infercnv_obj = CreateInfercnvObject(raw_counts_matrix=counts, annotations_file=annot, gene_order_file=gene, ref_group_names=ref.groups, chr_exclude=c("X","Y","MT"))
infercnv_obj = infercnv::run(infercnv_obj, cutoff=0.1, out_dir=Outdir, cluster_by_groups=FALSE, k_obs_groups=10, analysis_mode="subclusters", denoise=TRUE, HMM=TRUE, num_threads=4, resume_mode=T)

then, I got the following error during: 

STEP 18: Run Bayesian Network Model on HMM predicted CNV's
...
INFO [2019-11-20 11:17:08] plot_cnv_observation:Start
INFO [2019-11-20 11:17:08] Observation data size: Cells= 5494 Genes= 4218
INFO [2019-11-20 11:17:08] clustering observations via method: ward.D
Error in cp("(") : node stack overflow
Calls: <Anonymous> ... add.internal -> add.internal -> add.internal -> add.internal -> cp
Execution halted

The inputs should be OK, because I run these successfully by setting other parameters.

Here are my sessionINFO()

R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS/LAPACK: /software/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64_lin/libmkl_gf_lp64.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] grid      stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
 [1] ggplot2_3.2.1     infercnv_1.3.1    cowplot_1.0.0     reshape2_1.4.3   
 [5] dplyr_0.8.3       plyr_1.8.4        Seurat_3.1.1      R.utils_2.9.0    
 [9] R.oo_1.23.0       R.methodsS3_1.7.1 magrittr_1.5      colorout_1.2-1   

loaded via a namespace (and not attached):
  [1] TH.data_1.0-10              Rtsne_0.15                 
  [3] colorspace_1.4-1            modeltools_0.2-22          
  [5] ggridges_0.5.1              futile.logger_1.4.3        
  [7] XVector_0.26.0              GenomicRanges_1.38.0       
  [9] leiden_0.3.1                listenv_0.7.0              
 [11] npsurv_0.4-0                ggrepel_0.8.1              
 [13] mvtnorm_1.0-11              coin_1.3-1                 
 [15] codetools_0.2-16            splines_3.6.0              
 [17] doParallel_1.0.15           lsei_1.2-0                 
 [19] libcoin_1.0-5               zeallot_0.1.0              
 [21] jsonlite_1.6                ica_1.0-2                  
 [23] argparse_2.0.1              cluster_2.1.0              
 [25] rjags_4-10                  png_0.1-7                  
 [27] uwot_0.1.4                  sctransform_0.2.0          
 [29] compiler_3.6.0              httr_1.4.1                 
 [31] backports_1.1.5             assertthat_0.2.1           
 [33] Matrix_1.2-17               lazyeval_0.2.2             
 [35] limma_3.42.0                formatR_1.7                
 [37] htmltools_0.4.0             tools_3.6.0                
 [39] rsvd_1.0.2                  igraph_1.2.4.1             
 [41] coda_0.19-3                 gtable_0.3.0               
 [43] glue_1.3.1                  GenomeInfoDbData_1.2.2     
 [45] RANN_2.6.1                  Rcpp_1.0.3                 
 [47] Biobase_2.46.0              vctrs_0.2.0                
 [49] gdata_2.18.0                ape_5.3                    
 [51] nlme_3.1-142                iterators_1.0.12           
 [53] gbRd_0.4-11                 lmtest_0.9-37              
 [55] fastcluster_1.1.25          stringr_1.4.0              
 [57] globals_0.12.4              lifecycle_0.1.0            
 [59] irlba_2.3.3                 gtools_3.8.1               
 [61] future_1.15.0               edgeR_3.28.0               
 [63] MASS_7.3-51.4               zlibbioc_1.32.0            
 [65] zoo_1.8-6                   scales_1.1.0               
 [67] sandwich_2.5-1              parallel_3.6.0             
 [69] SummarizedExperiment_1.16.0 lambda.r_1.2.4             
 [71] RColorBrewer_1.1-2          SingleCellExperiment_1.8.0 
 [73] reticulate_1.13             pbapply_1.4-2              
 [75] gridExtra_2.3               reshape_0.8.8              
 [77] stringi_1.4.3               S4Vectors_0.24.0           
 [79] foreach_1.4.7               caTools_1.17.1.2           
 [81] BiocGenerics_0.32.0         BiocParallel_1.20.0        
 [83] bibtex_0.4.2                GenomeInfoDb_1.22.0        
 [85] matrixStats_0.55.0          Rdpack_0.11-0              
 [87] SDMTools_1.1-221.1          rlang_0.4.1                
 [89] pkgconfig_2.0.3             bitops_1.0-6               
 [91] lattice_0.20-38             ROCR_1.0-7                 
 [93] purrr_0.3.3                 htmlwidgets_1.5.1          
 [95] tidyselect_0.2.5            RcppAnnoy_0.0.14           
 [97] R6_2.4.1                    IRanges_2.20.0             
 [99] gplots_3.0.1.1              multcomp_1.4-10            
[101] DelayedArray_0.12.0         withr_2.1.2                
[103] pillar_1.4.2                findpython_1.0.5           
[105] fitdistrplus_1.0-14         survival_3.1-7             
[107] RCurl_1.95-4.12             tibble_2.1.3               
[109] future.apply_1.3.0          tsne_0.1-3                 
[111] crayon_1.3.4                futile.options_1.0.1       
[113] KernSmooth_2.23-16          plotly_4.9.1               
[115] locfit_1.5-9.1              data.table_1.12.6          
[117] metap_1.1                   digest_0.6.22              
[119] tidyr_1.0.0                 RcppParallel_4.4.4         
[121] stats4_3.6.0                munsell_0.5.0              
[123] viridisLite_0.3.0

So how to fix it?

GeorgescuC commented 4 years ago

Hi @QqQss ,

It sounds like the current options()$expressions value your R is configured with (probably the default of 5000) does not allow recursions deep enough to print/write the dendrogram since there are 5494 leaves in it. I would try changing it with options(expressions=10000) for example, then run infercnv again. I would also make sure the results look consistent with previous steps as having a dendrogram almost as deep as there are leaves in means it is probably not balanced, although most likely due to being an hclust of the Bayesian filtered predictions.

Regards, Christophe.

QqQss commented 4 years ago

OK, the program can continue to run now. Meanwhile, you said the dendrogram is not balanced. does it mean the result will be unreliable?

QqQss commented 4 years ago

@GeorgescuC Oops! The error still exists after setting options as you suggested. Actually, the program skipped several steps from step15 to 21 and then got the same error. Here are my commands:

options(expressions=500000)   ## avoid "node stack overflow" error
infercnv_obj = CreateInfercnvObject(raw_counts_matrix=counts, annotations_file=annot, gene_order_file=gene, ref_group_names=ref.groups, chr_exclude=c("X","Y","MT")) 
infercnv_obj = infercnv::run(infercnv_obj, cutoff=0.1, out_dir=Dir, cluster_by_groups=FALSE, k_obs_groups=10, analysis_mode="subclusters", denoise=TRUE, HMM=TRUE, num_threads=4, resume_mode=T)

and here are the log and error:

INFO [2019-11-21 14:25:50] ::order_reduce:Start.
INFO [2019-11-21 14:25:51] .order_reduce(): expr and order match.
INFO [2019-11-21 14:25:52] ::process_data:order_reduce:Reduction from positional data, new dimensions (r,c) = 28855,10897 Total=44665206 Min=0 Max=22733.
INFO [2019-11-21 14:25:52] num genes removed taking into account provided gene ordering list: 1009 = 3.49679431640963% removed.
INFO [2019-11-21 14:25:53] validating infercnv_obj
INFO [2019-11-21 14:25:53] ::process_data:Start
INFO [2019-11-21 14:25:53] Checking for saved results.
INFO [2019-11-21 14:25:53] Trying to reload from step 21
INFO [2019-11-21 14:25:56] Using backup from step 21
INFO [2019-11-21 14:25:56] Trying to reload from step 19
INFO [2019-11-21 14:25:59] Using backup HMM from step 19
INFO [2019-11-21 14:25:59] Trying to reload from step 18
INFO [2019-11-21 14:26:02] 

        STEP 1: incoming data

INFO [2019-11-21 14:26:02] 

        STEP 02: Removing lowly expressed genes

INFO [2019-11-21 14:26:02] 

        STEP 03: normalization by sequencing depth

INFO [2019-11-21 14:26:02] 

        STEP 04: log transformation of data

INFO [2019-11-21 14:26:02] 

        STEP 07: computing tumor subclusters via random_trees

INFO [2019-11-21 14:26:02] 

        STEP 08: removing average of reference data (before smoothing)

INFO [2019-11-21 14:26:02] 

        STEP 09: apply max centered expression threshold: 3

INFO [2019-11-21 14:26:02] 

        STEP 10: Smoothing data per cell by chromosome

INFO [2019-11-21 14:26:02] 

        STEP 11: re-centering data across chromosome after smoothing

INFO [2019-11-21 14:26:02] 

        STEP 12: removing average of reference data (after smoothing)

INFO [2019-11-21 14:26:02] 

        STEP 14: invert log2(FC) to FC

INFO [2019-11-21 14:26:02] 
        STEP 21: Denoising

INFO [2019-11-21 14:26:19] 

## Making the final infercnv heatmap ##
INFO [2019-11-21 14:26:19] ::plot_cnv:Start
INFO [2019-11-21 14:26:19] ::plot_cnv:Current data dimensions (r,c)=4218,10897 Total=137898318 Min=1 Max=5.
INFO [2019-11-21 14:26:20] ::plot_cnv:Depending on the size of the matrix this may take a moment.
INFO [2019-11-21 14:27:18] plot_cnv(): auto thresholding at: (-1.000000 , 3.000000)
INFO [2019-11-21 14:27:21] plot_cnv_observation:Start
INFO [2019-11-21 14:27:21] Observation data size: Cells= 5494 Genes= 4218
INFO [2019-11-21 14:27:21] clustering observations via method: ward.D

Error in cp("(") : node stack overflow
Calls: <Anonymous> ... add.internal -> add.internal -> add.internal -> add.internal -> cp
Execution halted
GeorgescuC commented 4 years ago

Hi @QqQss ,

There may have been an issue with reloading the results. The log says the HMM steps were seen as completed and so only the final residual expression plot needed to be rerun. The plotting step that fails should be that of the non HMM predictions, but the value within the matrix are "Min=1 Max=5", which corresponds to the HMM states, and not the residual expressions. I think you may need to rerun things with "resume_mode=F" or in a clean output folder.

It is still weird that you get the node stack overflow error after changing the expressions value. Could you check that it was properly updated by running options()$expressions? If you could also set options(error = function() traceback(2)) before restarting the run, it might give me more info about where the issue comes from.

Regards, Christophe.

XiaofeiSong commented 4 years ago

Hello, I am having the same issue. The code ran successfully with subclustering method qnorm but gave the 'node stack overflow' error with the random tree method. I've double checked the setting of the options(expression = 30000) by using options()$expressions.

Below is the error message. INFO [2020-05-15 04:42:18] -writing gene ordering info: result.subclustering.rt/17_HMM_predHMMi6.rand_trees.hmm_mode-subclusters.genes_used.dat INFO [2020-05-15 04:42:19] ::plot_cnv:Start INFO [2020-05-15 04:42:19] ::plot_cnv:Current data dimensions (r,c)=4865,24031 Total=353610585 Min=2 Max=5. INFO [2020-05-15 04:42:19] ::plot_cnv:Depending on the size of the matrix this may take a moment. INFO [2020-05-15 04:43:50] plot_cnv_observation:Start INFO [2020-05-15 04:43:50] Observation data size: Cells= 17139 Genes= 4865 Error: node stack overflow Error during wrapup: node stack overflow Error: no more error handlers available (recursive errors?); invoking 'abort' restart

Any suggestions? Thanks

GeorgescuC commented 4 years ago

Hi @XiaofeiSong ,

I see you have 17k cell, can you try increasing the C stack size limit outside of R before starting R and infercnv? Although you can increase the recursion limit of R, there is also a limitation outside of R. The command to run is ulimit -s unlimited.

Regards, Christophe.

AidenSb commented 4 years ago

Hi, @GeorgescuC I also have this issue in HMM-based CNV prediction step with one of my samples; I even split my seurat_object in half to reduce the size ( now running with ~8000 cells) I still got this error. the stack limit is in its max in my systems(16Mb). do you have any idea how I can solve this?

Cheers Aiden

GeorgescuC commented 3 years ago

Hi @AidenSb ,

Does this happen only on one of your samples? If yes, how big are the other samples that run fine? If you have multiple cell types, it may be worth trying to split your run by cell type and see if that works. I recently tried to diagnose some node stack overflow error in R when trying to combine dendrograms and their seem to be weird issues in how R internals handle them, so it is difficult to fix. Something else you could give a try is running this sample through a docker/singularity image and see if that changes anything.

Regards, Christophe.

Cristinex commented 1 year ago

I am also having node stack overflow issue after finishing runing step 17 and begining to plot. I tried to set ulimit -s unlimited, and also options(expressions=500000). But it still didn't work. What I got from another post was that the Total CNV number seems to be too large. Could you @GeorgescuC please help to solve the problem?

Here was my command. infercnv_obj = infercnv::run(infercnv_obj,cutoff=0.1,out_dir='/dir/',cluster_by_groups=T,denoise=T,HMM=T,num_threads=40)

Here was part of the report after I resume.

INFO [2023-02-08 10:47:58] Checking for saved results.
INFO [2023-02-08 10:47:58] Trying to reload from step 17
INFO [2023-02-08 10:48:38] Using backup HMM from step 17
INFO [2023-02-08 10:48:38] Trying to reload from step 15
INFO [2023-02-08 10:49:51] Using backup from step 15
INFO [2023-02-08 10:49:51]

        STEP 1: incoming data

INFO [2023-02-08 10:49:51]

        STEP 02: Removing lowly expressed genes

INFO [2023-02-08 10:49:51]

        STEP 03: normalization by sequencing depth

INFO [2023-02-08 10:49:51]

        STEP 04: log transformation of data

INFO [2023-02-08 10:49:51]

        STEP 08: removing average of reference data (before smoothing)

INFO [2023-02-08 10:49:51]

        STEP 09: apply max centered expression threshold: 3

INFO [2023-02-08 10:49:51]

        STEP 10: Smoothing data per cell by chromosome

INFO [2023-02-08 10:49:51]

        STEP 11: re-centering data across chromosome after smoothing

INFO [2023-02-08 10:49:51]

        STEP 12: removing average of reference data (after smoothing)

INFO [2023-02-08 10:49:51]

        STEP 14: invert log2(FC) to FC

INFO [2023-02-08 10:49:51]

        STEP 15: computing tumor subclusters via leiden

INFO [2023-02-08 10:50:43]

        STEP 18: Run Bayesian Network Model on HMM predicted CNVs

INFO [2023-02-08 10:50:44] Creating the following Directory:  /dir/BayesNetOutput.HMMi6.leiden.hmm_mode-subclusters
INFO [2023-02-08 10:50:44] Initializing new MCM InferCNV Object.
INFO [2023-02-08 10:50:44] validating infercnv_obj
INFO [2023-02-08 10:51:37] Total CNV's:  368267
INFO [2023-02-08 10:51:37] Loading BUGS Model.
GeorgescuC commented 1 year ago

Hi @Cristinex ,

Have you restarted R after setting ulimit? And if you print current ulimit values, is the stack size actually changed? I recently realized trying to change the ulimit within a docker image for example fails silently and you need to set the option when starting the image. If you are running on a server, you might also have limits when trying to change the option.

As for the number of CNVs, it does indeed look very high. If this is on the same dataset as your other question with ~100k cells, that would be on average more than 3 CNVs per single cell, even though CNVs are defined for (sub)clusters of cells. The main issue might be that the subclusters are too fragmented like in https://github.com/broadinstitute/infercnv/issues/496 .

I would run the following code to try to inspect subclusters and see if fragmentation is the issue. If you encounter the stack limit error while plotting, at least checking the length() call output before the plot_cnv() call should indicate how many subclusters are generated for the observations.

subcluster_obj = infercnv_obj
subcluster_obj@reference_grouped_cell_indices = list()
for (grp in names(infercnv_obj@reference_grouped_cell_indices)) {
    for (grp2 in names(infercnv_obj@tumor_subclusters$subclusters[[grp]])) {
        subcluster_obj@reference_grouped_cell_indices[[grp2]] = infercnv_obj@tumor_subclusters$subclusters[[grp]][[grp2]]
    }
}

subcluster_obj@observation_grouped_cell_indices = list()
for (grp in names(infercnv_obj@tumor_subclusters$subclusters)) {
    for (grp2 in names(infercnv_obj@tumor_subclusters$subclusters[[grp]])) {
        subcluster_obj@observation_grouped_cell_indices[[grp2]] = infercnv_obj@tumor_subclusters$subclusters[[grp]][[grp2]]
    }
}

subcluster_obj@tumor_subclusters = NULL

length(subcluster_obj@observation_grouped_cell_indices)

plot_cnv(subcluster_obj,
         cluster_by_groups=TRUE,
         output_filename = output_filename,
         out_dir=out_dir,
         write_expr_matrix=FALSE)

Regards, Christophe.