PHATE causes R to abort when n.jobs is low or C stack error when n.jobs is high

[rschauner]

Describe the bug When running PHATE R suddenly aborts before any messages are printed to the console.

DefaultAssay(seurat) <- "SCT"
seurat <- RunPCA(seurat)
seurat <- RunHarmony(seurat,
                    c("seq_batch", "sort_batch", "patient_id"),
                    assay.use = "SCT",
                    reduction.save = "sct_harmony",
                    max.iter.cluster = 30)
seurat <- RunUMAP(seurat,
                  dims = 1:30,
                  reduction = "sct_harmony")
seurat <- RunPHATE.Seurat(seurat, reduction = "pca", n.jobs = 8)

The function RunPHATE is from Scott's branch of Seurat (but I did not install that branch, just copied the code into my R environment). I tried with reduction = "sct_harmony" and n.jobs = 1 and n.jobs = 4. When n.jobs = 8 I get a C stack memory usage error around the KNN step, and for n.jobs = 1 and n.jobs = 4 R just aborts.

C stack error:

Calculating PHATE...
  Running PHATE on 2700 observations and 50 variables.
  Calculating graph and diffusion operator...
    Calculating KNN search...
Error: C stack usage  17587261002064 is too close to the limit

To Reproduce run above code on the 3k PBMC data from 10x (loaded through SeuratData) with no filtering:

data("pbmc3k")
seurat <- pbmc3k
seurat <- NormalizeData(seurat) 
seurat <- FindVariableFeatures(seurat) 
seurat <- ScaleData(seurat)
seurat <- RunPCA(seurat)
seurat <- RunPHATE.Seurat(seurat, reduction = "pca", n.jobs = 1)

Running with n.jobs = 4 replicates C stack error. Here are the needed functions to source: RunPHATE.txt

Expected behavior R to continue running PHATE or signal an error

Actual behavior R session aborts with no errors

System information:

Output of phate.__version__: '1.0.4'

Output of pd.show_versions():

``` INSTALLED VERSIONS ------------------ commit : b5958ee1999e9aead1938c0bba2b674378807b3d python : 3.6.12.final.0 python-bits : 64 OS : Darwin OS-release : 19.6.0 Version : Darwin Kernel Version 19.6.0: Tue Nov 10 00:10:30 PST 2020; root:xnu-6153.141.10~1/RELEASE_X86_64 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 1.1.5 numpy : 1.19.5 pytz : 2020.5 dateutil : 2.8.1 pip : 20.3.3 setuptools : 49.6.0.post20210108 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : None IPython : None pandas_datareader: None bs4 : None bottleneck : None fsspec : None fastparquet : None gcsfs : None matplotlib : 3.3.3 numexpr : 2.7.2 odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pytables : None pyxlsb : None s3fs : None scipy : 1.5.3 sqlalchemy : None tables : 3.6.1 tabulate : None xarray : None xlrd : None xlwt : None numba : 0.52.0 ```

Output of sessionInfo():

``` R version 4.0.3 (2020-10-10) Platform: x86_64-apple-darwin17.0 (64-bit) Running under: macOS Catalina 10.15.7 Matrix products: default BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib locale: [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8 attached base packages: [1] stats graphics grDevices datasets utils methods base other attached packages: [1] forcats_0.5.0 stringr_1.4.0 dplyr_1.0.3 purrr_0.3.4 readr_1.4.0 tidyr_1.1.2 [7] tibble_3.0.5 ggplot2_3.3.3 tidyverse_1.3.0 magrittr_2.0.1 future_1.21.0 SeuratWrappers_0.3.0 [13] harmony_1.0 Rcpp_1.0.6 readxl_1.3.1 here_1.0.1 Seurat_3.2.3 loaded via a namespace (and not attached): [1] backports_1.2.1 plyr_1.8.6 igraph_1.2.6 lazyeval_0.2.2 [5] splines_4.0.3 listenv_0.8.0 scattermore_0.7 GenomeInfoDb_1.24.2 [9] digest_0.6.27 htmltools_0.5.1.1 viridis_0.5.1 fansi_0.4.2 [13] memoise_1.1.0 phateR_1.0.4 tensor_1.5 cluster_2.1.0 [17] ROCR_1.0-11 remotes_2.2.0 globals_0.14.0 modelr_0.1.8 [21] matrixStats_0.57.0 colorspace_2.0-0 rvest_0.3.6 ggrepel_0.9.1 [25] haven_2.3.1 crayon_1.3.4 RCurl_1.98-1.2 jsonlite_1.7.2 [29] spatstat_1.64-1 spatstat.data_1.7-0 survival_3.2-7 zoo_1.8-8 [33] glue_1.4.2 polyclip_1.10-0 gtable_0.3.0 zlibbioc_1.34.0 [37] XVector_0.28.0 leiden_0.3.6 DelayedArray_0.14.1 future.apply_1.7.0 [41] SingleCellExperiment_1.10.1 BiocGenerics_0.34.0 abind_1.4-5 scales_1.1.1 [45] DBI_1.1.1 miniUI_0.1.1.1 viridisLite_0.3.0 xtable_1.8-4 [49] reticulate_1.18 rsvd_1.0.3 stats4_4.0.3 htmlwidgets_1.5.3 [53] httr_1.4.2 RColorBrewer_1.1-2 ellipsis_0.3.1 ica_1.0-2 [57] pkgconfig_2.0.3 uwot_0.1.10 dbplyr_2.0.0 deldir_0.2-9 [61] tidyselect_1.1.0 rlang_0.4.10 reshape2_1.4.4 later_1.1.0.1 [65] munsell_0.5.0 cellranger_1.1.0 tools_4.0.3 cli_2.2.0 [69] generics_0.1.0 broom_0.7.3 ggridges_0.5.3 fastmap_1.1.0 [73] goftest_1.2-2 fs_1.5.0 fitdistrplus_1.1-3 RANN_2.6.1 [77] pbapply_1.4-3 nlme_3.1-151 mime_0.9 monocle3_0.2.3.0 [81] xml2_1.3.2 compiler_4.0.3 rstudioapi_0.13 plotly_4.9.3 [85] png_0.1-7 spatstat.utils_2.0-0 reprex_0.3.0 stringi_1.5.3 [89] lattice_0.20-41 Matrix_1.3-2 vctrs_0.3.6 pillar_1.4.7 [93] lifecycle_0.2.0 BiocManager_1.30.10 lmtest_0.9-38 RcppAnnoy_0.0.18 [97] data.table_1.13.6 cowplot_1.1.1 bitops_1.0-6 irlba_2.3.3 [101] httpuv_1.5.5 patchwork_1.1.1 GenomicRanges_1.40.0 R6_2.5.0 [105] promises_1.1.1 renv_0.12.5 KernSmooth_2.23-18 gridExtra_2.3 [109] IRanges_2.22.2 parallelly_1.23.0 codetools_0.2-18 MASS_7.3-53 [113] assertthat_0.2.1 SummarizedExperiment_1.18.2 rprojroot_2.0.2 withr_2.4.0 [117] sctransform_0.3.2 S4Vectors_0.26.1 GenomeInfoDbData_1.2.3 mgcv_1.8-33 [121] parallel_4.0.3 hms_1.0.0 grid_4.0.3 rpart_4.1-15 [125] Rtsne_0.15 Biobase_2.48.0 shiny_1.5.0 lubridate_1.7.9.2 ```

Output of reticulate::py_discover_config(required_module = "phate"):

``` python: /Users/schaunr/Documents/Lab/clonal_evolution/AML_clonal_evolution/renv/python/condaenvs/renv-python/bin/python libpython: /Users/schaunr/Documents/Lab/clonal_evolution/AML_clonal_evolution/renv/python/condaenvs/renv-python/lib/libpython3.6m.dylib pythonhome: /Users/schaunr/Documents/Lab/clonal_evolution/AML_clonal_evolution/renv/python/condaenvs/renv-python:/Users/schaunr/Documents/Lab/clonal_evolution/AML_clonal_evolution/renv/python/condaenvs/renv-python version: 3.6.12 | packaged by conda-forge | (default, Dec 9 2020, 00:24:39) [GCC Clang 11.0.0] numpy: /Users/schaunr/Documents/Lab/clonal_evolution/AML_clonal_evolution/renv/python/condaenvs/renv-python/lib/python3.6/site-packages/numpy numpy_version: 1.19.5 phate: /Users/schaunr/Documents/Lab/clonal_evolution/AML_clonal_evolution/renv/python/condaenvs/renv-python/lib/python3.6/site-packages/phate NOTE: Python version was forced by RETICULATE_PYTHON ```

Output of phateR::check_pyphate_version():

``` Error: object of type 'closure' is not subsettable ``` I double checked this + tried copying and typing it myself with autocomplete the code

Additional context Seurat Object:

An object of class Seurat 
76235 features across 284987 samples within 4 assays 
Active assay: ADT (20 features, 20 variable features)
 3 other assays present: RNA, HTO, SCT

[rschauner]

I also would like to mention that it has worked before and I was able to create plots when I was testing out the code.

[scottgigante]

Hi @rschauner , sorry for the delayed response on this. The error

Error: object of type 'closure' is not subsettable

indicates to me that your phateR module is out of date. Could you update it and try again?

[rschauner]

I ended up giving up on PHATE for awhile (because of problems with renv and reticulate), but tried it again on Friday on a new computer (so new install) and it seems to work.