saketkc
commented
1 year ago With 20 datasets I would recommend using RPCA or reference based integration. This vignette offers some tips that should be helpful!
Closed zorglubz-coder closed 1 year ago
saketkc
commented
1 year ago With 20 datasets I would recommend using RPCA or reference based integration. This vignette offers some tips that should be helpful!
zorglubz-coder
commented
1 year ago Thanks, I made four of my samples reference, and it is working great!
hoonghim
commented
1 year ago Dear @saketkc ,
Hello, I am also having the same problem after following the vignette you recommended.
I tried reference based integration after running SCTransform.
The same error still occurs during the IntegrateData step.
I am utilizing ~400,000 cells from 86 samples (25 samples are from public normal dataset).
Could the dataset being too large have something to do with this problem?
Here are my codes and error messages.
# Read Seurat object
x.obj.filter <- readRDS(file = x.file)
# Run FindVariableFeatures
x.obj.filter <- FindVariableFeatures(object = x.obj.filter, selection.method = "vst")
# Regress out cell cycle scores
s.genes <- cc.genes$s.genes
g2m.genes <- cc.genes$g2m.genes
x.obj.filter <- CellCycleScoring(x.obj.filter, s.features = s.genes, g2m.features = g2m.genes, set.ident = T)
x.obj.filter <- ScaleData(object = x.obj.filter, features = rownames(x.obj.filter), vars.to.regress = c("nCount_RNA", "percent.mt", "S.Score", "G2M.Score"))
# Perform linear dimensional reduction
x.obj.filter <- RunPCA(object = x.obj.filter, features = VariableFeatures(object = x.obj.filter), npcs = 50)
#######
# Run SCTransform and integrate data using CCA
# Split Seurat object by sample
rds.list <- SplitObject(x.obj.filter, split.by = "orig.ident")
# Run SCTransform
for (i in names(rds.list)) {
print(i)
rds.list[[i]] <- SCTransform(rds.list[[i]], verbose = FALSE, vars.to.regress = c("nCount_RNA", "percent.mt", "S.Score", "G2M.Score"), method="glmGamPoi")
}
rds.features <- SelectIntegrationFeatures(object.list = rds.list, nfeatures = 3000)
rds.list <- PrepSCTIntegration(object.list = rds.list, anchor.features = rds.features)
# Set reference dataset using normal sample indices
reference_dataset <- which(!grepl(pattern = "DIS|^01-D", names(rds.list)))
# Run FindIntegrationAnchors using reference dataset
rds.anchors <- FindIntegrationAnchors(object.list = rds.list,
normalization.method = "SCT",
anchor.features = rds.features,
reference = reference_dataset)
# Run IntegrateData
rds.integrated <- IntegrateData(anchorset = rds.anchors, normalization.method = "SCT")
Integrating data
Merging dataset 8 into 10 11 68 78 77 61 65
Extracting anchors for merged samples
Finding integration vectors
Finding integration vector weights
0% 10 20 30 40 50 60 70 80 90 100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Integrating data
Merging dataset 66 into 69 72 73 74 64
Extracting anchors for merged samples
Finding integration vectors
Finding integration vector weights
0% 10 20 30 40 50 60 70 80 90 100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Integrating data
Merging dataset 70 79 into 67 75 62
Extracting anchors for merged samples
Finding integration vectors
Finding integration vector weights
0% 10 20 30 40 50 60 70 80 90 100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Integrating data
Merging dataset 9 into 10 11 68 78 77 61 65 8
Extracting anchors for merged samples
Finding integration vectors
Finding integration vector weights
0% 10 20 30 40 50 60 70 80 90 100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Integrating data
Merging dataset 76 into 69 72 73 74 64 66
Extracting anchors for merged samples
Finding integration vectors
Finding integration vector weights
0% 10 20 30 40 50 60 70 80 90 100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Integrating data
Merging dataset 67 75 62 70 79 into 69 72 73 74 64 66 76
Extracting anchors for merged samples
Finding integration vectors
Finding integration vector weights
0% 10 20 30 40 50 60 70 80 90 100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Integrating data
Merging dataset 10 11 68 78 77 61 65 8 9 into 69 72 73 74 64 66 76 67 75 62 70 79
Extracting anchors for merged samples
Finding integration vectors
Error in .T2C(newTMat(i = c(ij1[, 1], ij2[, 1]), j = c(ij1[, 2], ij2[, :
unable to coerce from TsparseMatrix to [CR]sparseMatrixwhen length of 'i' slot exceeds 2^31-1
In addition: There were 50 or more warnings (use warnings() to see the first 50)
> reference_dataset
[1] 8 9 10 11 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79
> warnings()
Warning messages:
1: Detected creation of a 'multiprocess' future. Strategy 'multiprocess' is deprecated in future (>= 1.20.0) [2020-10-30]. Instead, explicitly specify either 'multisession' (recommended) or 'multicore'. In the current R session, 'multiprocess' equals 'multicore'.
2: Detected creation of a 'multiprocess' future. Strategy 'multiprocess' is deprecated in future (>= 1.20.0) [2020-10-30]. Instead, explicitly specify either 'multisession' (recommended) or 'multicore'. In the current R session, 'multiprocess' equals 'multicore'.
...
50: Detected creation of a 'multiprocess' future. Strategy 'multiprocess' is deprecated in future (>= 1.20.0) [2020-10-30]. Instead, explicitly specify either 'multisession' (recommended) or 'multicore'. In the current R session, 'multiprocess' equals 'multicore'.
Here is my session info
R version 4.0.5 (2021-03-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.1 LTS
Matrix products: default
BLAS: /BiO/Live/hoonkim/.Renv/versions/R-4.0.5/lib/R/lib/libRblas.so
LAPACK: /BiO/Live/hoonkim/.Renv/versions/R-4.0.5/lib/R/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] future_1.30.0 stringr_1.5.0 patchwork_1.1.2 ggplot2_3.4.0
[5] SeuratData_0.2.2 harmony_0.1.1 Rcpp_1.0.9 Cairo_1.6-0
[9] doMC_1.3.8 iterators_1.0.14 foreach_1.5.2 RColorBrewer_1.1-3
[13] SeuratObject_4.1.3 Seurat_4.3.0
loaded via a namespace (and not attached):
[1] nlme_3.1-152 matrixStats_0.63.0 spatstat.sparse_3.0-0
[4] RcppAnnoy_0.0.20 httr_1.4.4 sctransform_0.3.5
[7] tools_4.0.5 utf8_1.2.2 R6_2.5.1
[10] irlba_2.3.5.1 KernSmooth_2.23-18 uwot_0.1.14
[13] DBI_1.1.3 lazyeval_0.2.2 colorspace_2.0-3
[16] withr_2.5.0 sp_1.5-1 tidyselect_1.2.0
[19] gridExtra_2.3 compiler_4.0.5 progressr_0.12.0
[22] cli_3.5.0 spatstat.explore_3.0-5 plotly_4.10.1
[25] scales_1.2.1 lmtest_0.9-40 spatstat.data_3.0-0
[28] ggridges_0.5.4 pbapply_1.6-0 rappdirs_0.3.3
[31] goftest_1.2-3 digest_0.6.31 spatstat.utils_3.0-1
[34] pkgconfig_2.0.3 htmltools_0.5.4 parallelly_1.33.0
[37] fastmap_1.1.0 htmlwidgets_1.6.0 rlang_1.0.6
[40] shiny_1.7.4 generics_0.1.3 zoo_1.8-11
[43] jsonlite_1.8.4 spatstat.random_3.0-1 ica_1.0-3
[46] dplyr_1.0.10 magrittr_2.0.3 Matrix_1.5-3
[49] munsell_0.5.0 fansi_1.0.3 abind_1.4-5
[52] reticulate_1.26 lifecycle_1.0.3 stringi_1.7.8
[55] MASS_7.3-53.1 Rtsne_0.16 plyr_1.8.8
[58] grid_4.0.5 listenv_0.9.0 promises_1.2.0.1
[61] ggrepel_0.9.2 crayon_1.5.2 deldir_1.0-6
[64] miniUI_0.1.1.1 lattice_0.20-41 cowplot_1.1.1
[67] splines_4.0.5 tensor_1.5 pillar_1.8.1
[70] igraph_1.3.5 spatstat.geom_3.0-3 future.apply_1.10.0
[73] reshape2_1.4.4 codetools_0.2-18 leiden_0.4.3
[76] glue_1.6.2 data.table_1.14.6 png_0.1-8
[79] vctrs_0.5.1 httpuv_1.6.7 polyclip_1.10-4
[82] gtable_0.3.1 RANN_2.6.1 purrr_1.0.0
[85] tidyr_1.2.1 scattermore_0.8 assertthat_0.2.1
[88] mime_0.12 xtable_1.8-4 later_1.3.0
[91] survival_3.2-10 viridisLite_0.4.1 tibble_3.1.8
[94] cluster_2.1.1 globals_0.16.2 fitdistrplus_1.1-8
[97] ellipsis_0.3.2 ROCR_1.0-11
Any suggestions would be of great help.
Sincerely,
Seunghoon
zorglubz-coder
commented
1 year ago Hi @hoonghim, How many datasets are you using for your reference? I only used four, and the issue is if you use too many in the FindIntegrationAnchors, then it is too much for IntegrateData. I don't knwo the max number of datasets you can use, but you should try running only on a few of them (less than 5) and see if this is woking. Then you can try to repeat with more and more. But too many is the issue (in my opinion as a Seurat user)
luzzhou
commented
4 months ago Dear @saketkc ,
Hello, I am also having the same problem after following the vignette you recommended.
I tried reference based integration after running SCTransform.
The same error still occurs during the IntegrateData step.
I am utilizing ~400,000 cells from 86 samples (25 samples are from public normal dataset).
Could the dataset being too large have something to do with this problem?
Here are my codes and error messages.
# Read Seurat object x.obj.filter <- readRDS(file = x.file) # Run FindVariableFeatures x.obj.filter <- FindVariableFeatures(object = x.obj.filter, selection.method = "vst") # Regress out cell cycle scores s.genes <- cc.genes$s.genes g2m.genes <- cc.genes$g2m.genes x.obj.filter <- CellCycleScoring(x.obj.filter, s.features = s.genes, g2m.features = g2m.genes, set.ident = T) x.obj.filter <- ScaleData(object = x.obj.filter, features = rownames(x.obj.filter), vars.to.regress = c("nCount_RNA", "percent.mt", "S.Score", "G2M.Score")) # Perform linear dimensional reduction x.obj.filter <- RunPCA(object = x.obj.filter, features = VariableFeatures(object = x.obj.filter), npcs = 50) ####### # Run SCTransform and integrate data using CCA # Split Seurat object by sample rds.list <- SplitObject(x.obj.filter, split.by = "orig.ident") # Run SCTransform for (i in names(rds.list)) { print(i) rds.list[[i]] <- SCTransform(rds.list[[i]], verbose = FALSE, vars.to.regress = c("nCount_RNA", "percent.mt", "S.Score", "G2M.Score"), method="glmGamPoi") } rds.features <- SelectIntegrationFeatures(object.list = rds.list, nfeatures = 3000) rds.list <- PrepSCTIntegration(object.list = rds.list, anchor.features = rds.features) # Set reference dataset using normal sample indices reference_dataset <- which(!grepl(pattern = "DIS|^01-D", names(rds.list))) # Run FindIntegrationAnchors using reference dataset rds.anchors <- FindIntegrationAnchors(object.list = rds.list, normalization.method = "SCT", anchor.features = rds.features, reference = reference_dataset) # Run IntegrateData rds.integrated <- IntegrateData(anchorset = rds.anchors, normalization.method = "SCT") Integrating data Merging dataset 8 into 10 11 68 78 77 61 65 Extracting anchors for merged samples Finding integration vectors Finding integration vector weights 0% 10 20 30 40 50 60 70 80 90 100% [----|----|----|----|----|----|----|----|----|----| **************************************************| Integrating data Merging dataset 66 into 69 72 73 74 64 Extracting anchors for merged samples Finding integration vectors Finding integration vector weights 0% 10 20 30 40 50 60 70 80 90 100% [----|----|----|----|----|----|----|----|----|----| **************************************************| Integrating data Merging dataset 70 79 into 67 75 62 Extracting anchors for merged samples Finding integration vectors Finding integration vector weights 0% 10 20 30 40 50 60 70 80 90 100% [----|----|----|----|----|----|----|----|----|----| **************************************************| Integrating data Merging dataset 9 into 10 11 68 78 77 61 65 8 Extracting anchors for merged samples Finding integration vectors Finding integration vector weights 0% 10 20 30 40 50 60 70 80 90 100% [----|----|----|----|----|----|----|----|----|----| **************************************************| Integrating data Merging dataset 76 into 69 72 73 74 64 66 Extracting anchors for merged samples Finding integration vectors Finding integration vector weights 0% 10 20 30 40 50 60 70 80 90 100% [----|----|----|----|----|----|----|----|----|----| **************************************************| Integrating data Merging dataset 67 75 62 70 79 into 69 72 73 74 64 66 76 Extracting anchors for merged samples Finding integration vectors Finding integration vector weights 0% 10 20 30 40 50 60 70 80 90 100% [----|----|----|----|----|----|----|----|----|----| **************************************************| Integrating data Merging dataset 10 11 68 78 77 61 65 8 9 into 69 72 73 74 64 66 76 67 75 62 70 79 Extracting anchors for merged samples Finding integration vectors Error in .T2C(newTMat(i = c(ij1[, 1], ij2[, 1]), j = c(ij1[, 2], ij2[, : unable to coerce from TsparseMatrix to [CR]sparseMatrixwhen length of 'i' slot exceeds 2^31-1 In addition: There were 50 or more warnings (use warnings() to see the first 50) > reference_dataset [1] 8 9 10 11 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 > warnings() Warning messages: 1: Detected creation of a 'multiprocess' future. Strategy 'multiprocess' is deprecated in future (>= 1.20.0) [2020-10-30]. Instead, explicitly specify either 'multisession' (recommended) or 'multicore'. In the current R session, 'multiprocess' equals 'multicore'. 2: Detected creation of a 'multiprocess' future. Strategy 'multiprocess' is deprecated in future (>= 1.20.0) [2020-10-30]. Instead, explicitly specify either 'multisession' (recommended) or 'multicore'. In the current R session, 'multiprocess' equals 'multicore'. ... 50: Detected creation of a 'multiprocess' future. Strategy 'multiprocess' is deprecated in future (>= 1.20.0) [2020-10-30]. Instead, explicitly specify either 'multisession' (recommended) or 'multicore'. In the current R session, 'multiprocess' equals 'multicore'.Here is my session info
R version 4.0.5 (2021-03-31) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 22.04.1 LTS Matrix products: default BLAS: /BiO/Live/hoonkim/.Renv/versions/R-4.0.5/lib/R/lib/libRblas.so LAPACK: /BiO/Live/hoonkim/.Renv/versions/R-4.0.5/lib/R/lib/libRlapack.so locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 [7] LC_PAPER=en_US.UTF-8 LC_NAME=C [9] LC_ADDRESS=C LC_TELEPHONE=C [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C attached base packages: [1] parallel stats graphics grDevices utils datasets methods [8] base other attached packages: [1] future_1.30.0 stringr_1.5.0 patchwork_1.1.2 ggplot2_3.4.0 [5] SeuratData_0.2.2 harmony_0.1.1 Rcpp_1.0.9 Cairo_1.6-0 [9] doMC_1.3.8 iterators_1.0.14 foreach_1.5.2 RColorBrewer_1.1-3 [13] SeuratObject_4.1.3 Seurat_4.3.0 loaded via a namespace (and not attached): [1] nlme_3.1-152 matrixStats_0.63.0 spatstat.sparse_3.0-0 [4] RcppAnnoy_0.0.20 httr_1.4.4 sctransform_0.3.5 [7] tools_4.0.5 utf8_1.2.2 R6_2.5.1 [10] irlba_2.3.5.1 KernSmooth_2.23-18 uwot_0.1.14 [13] DBI_1.1.3 lazyeval_0.2.2 colorspace_2.0-3 [16] withr_2.5.0 sp_1.5-1 tidyselect_1.2.0 [19] gridExtra_2.3 compiler_4.0.5 progressr_0.12.0 [22] cli_3.5.0 spatstat.explore_3.0-5 plotly_4.10.1 [25] scales_1.2.1 lmtest_0.9-40 spatstat.data_3.0-0 [28] ggridges_0.5.4 pbapply_1.6-0 rappdirs_0.3.3 [31] goftest_1.2-3 digest_0.6.31 spatstat.utils_3.0-1 [34] pkgconfig_2.0.3 htmltools_0.5.4 parallelly_1.33.0 [37] fastmap_1.1.0 htmlwidgets_1.6.0 rlang_1.0.6 [40] shiny_1.7.4 generics_0.1.3 zoo_1.8-11 [43] jsonlite_1.8.4 spatstat.random_3.0-1 ica_1.0-3 [46] dplyr_1.0.10 magrittr_2.0.3 Matrix_1.5-3 [49] munsell_0.5.0 fansi_1.0.3 abind_1.4-5 [52] reticulate_1.26 lifecycle_1.0.3 stringi_1.7.8 [55] MASS_7.3-53.1 Rtsne_0.16 plyr_1.8.8 [58] grid_4.0.5 listenv_0.9.0 promises_1.2.0.1 [61] ggrepel_0.9.2 crayon_1.5.2 deldir_1.0-6 [64] miniUI_0.1.1.1 lattice_0.20-41 cowplot_1.1.1 [67] splines_4.0.5 tensor_1.5 pillar_1.8.1 [70] igraph_1.3.5 spatstat.geom_3.0-3 future.apply_1.10.0 [73] reshape2_1.4.4 codetools_0.2-18 leiden_0.4.3 [76] glue_1.6.2 data.table_1.14.6 png_0.1-8 [79] vctrs_0.5.1 httpuv_1.6.7 polyclip_1.10-4 [82] gtable_0.3.1 RANN_2.6.1 purrr_1.0.0 [85] tidyr_1.2.1 scattermore_0.8 assertthat_0.2.1 [88] mime_0.12 xtable_1.8-4 later_1.3.0 [91] survival_3.2-10 viridisLite_0.4.1 tibble_3.1.8 [94] cluster_2.1.1 globals_0.16.2 fitdistrplus_1.1-8 [97] ellipsis_0.3.2 ROCR_1.0-11
Any suggestions would be of great help.
Sincerely,
Seunghoon
Hi @hoonghim, have you solved this problem? I encountered the same error as you. I have 70 samples that need to be integrated đŸ˜©
Error in .M2C(newTMat(i = c(ij1[, 1], ij2[, 1]), j = c(ij1[, 2], ij2[, : unable to aggregate TsparseMatrix with 'i' and 'j' slots of length exceeding 2^31-1
Hi, Thank you for providing this amazing tool!
I am trying to integrate 20 samples, for a total of 85,000 cells. I am following the "Performing integration on datasets normalized with SCTransform" from https://satijalab.org/seurat/articles/integration_rpca.html almost to the letter. I am just using the v2 of SCTransform.
At first the process crashed the computer, I am now running it with 500 Gb of memory.
My issue now is that the function IntegrateData ends with this error message:
I have looked around, and I could not find this error message in other issues. Is there a workaround for me to integrate my data?
Here is my session info