search

abelson-lab / scATOMIC

Pan-Cancer Single Cell Classifier
MIT License
57 stars 5 forks source link

Input matrix contains unexpressed genes. #6

Closed u2sj closed 1 year ago

u2sj commented 1 year ago

I performed the code as workflow :lung_cancer_demo_data <- demo_lung_data

Preprocessing dataset

pct_mt <- colSums(lung_cancer_demo_data[grep("^MT-", row.names(lung_cancer_demo_data)),])/colSums(lung_cancer_demo_data) * 100 nFeatureRNA <- colSums(lung_cancer_demo_data > 0) lung_cancer_demo_data <- lung_cancer_demo_data[, names(which(pct_mt < 15))] lung_cancer_demo_data <- lung_cancer_demo_data[, intersect(names(which(nFeatureRNA > 500)), colnames(lung_cancer_demo_data))]

Running scATOMIC

cell_predictions <- run_scATOMIC(lung_cancer_demo_data,mc.cores = (parallel::detectCores() - 1))

However: there is a warning: /home/ubuntu/anaconda3/envs/r-reticulate/lib/python3.9/site-packages/magic/magic.py:425: UserWarning: Input matrix contains unexpressed genes. Please remove them prior to running MAGIC. warnings.warn(

inofechm commented 1 year ago

Thank you for using scATOMIC! This warning is normal, it comes up from the RMagic package, you can ignore it. Was the cell_predictions object able to be generated? Please let me know and I will close the issue.

u2sj commented 1 year ago

Thank you for your response.！ I don't think it's working properly as it has been running for 20 hours and still hasn't produced any files. If it runs successfully, I hope to apply it to my own data analysis. Also, I wonder how much data can be analyzed with 256GB memory? Thank you!

inofechm commented 1 year ago

the workflow in the tutorial should only take 5-10 minutes to run with less than 16 GB of RAM needed. I am not sure why it is stalled on your machine, I would restart the R session and try again. can you try running it with mc.cores set to 1? I am wondering if you are using too many parallel processes. if you are still having issue could you please share the output of your SessionInfo?

In terms of how much data can be analysed, scATOMIC should be used on each individual patient sample data so there are not many datasets that would require 256 GB of memory, in my hands datasets of 10,000-20,000 cells only require 15-20GB of memory.

u2sj commented 1 year ago

Thank you for your response. I have run the tutorial code after restart my computer as below: ...

library(cutoff) library(copykat) library(ggplot2) lung_cancer_demo_data <- demo_lung_data

Preprocessing dataset

pct_mt <- colSums(lung_cancer_demo_data[grep("^MT-", row.names(lung_cancer_demo_data)),])/colSums(lung_cancer_demo_data) * 100 nFeatureRNA <- colSums(lung_cancer_demo_data > 0) lung_cancer_demo_data <- lung_cancer_demo_data[, names(which(pct_mt < 15))] lung_cancer_demo_data <- lung_cancer_demo_data[, intersect(names(which(nFeatureRNA > 500)), colnames(lung_cancer_demo_data))]

Running scATOMIC

cell_predictions <- run_scATOMIC(lung_cancer_demo_data,mc.cores = (parallel::detectCores() - 1)) [1] "Starting Layer 1" /home/ubuntu/anaconda3/envs/r-reticulate/lib/python3.9/site-packages/magic/magic.py:425: UserWarning: Input matrix contains unexpressed genes. Please remove them prior to running MAGIC. warnings.warn(

─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────── setting value version R version 4.2.3 (2023-03-15) os Ubuntu 20.04.6 LTS system x86_64, linux-gnu ui RStudio language zh_CN:en collate zh_CN.UTF-8 ctype zh_CN.UTF-8 tz Asia/Shanghai date 2023-04-28 rstudio 2023.03.0+386 Cherry Blossom (desktop) pandoc NA

─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── package version date (UTC) lib source abind 1.4-5 2016-07-21 [1] CRAN (R 4.2.3) backports 1.4.1 2021-12-13 [1] CRAN (R 4.2.3) BiocManager 1.30.20 2023-02-24 [1] CRAN (R 4.2.3) broom 1.0.4 2023-03-11 [1] CRAN (R 4.2.3) cachem 1.0.7 2023-02-24 [1] CRAN (R 4.2.3) callr 3.7.3 2022-11-02 [1] CRAN (R 4.2.3) car 3.1-2 2023-03-30 [1] CRAN (R 4.2.3) carData 3.0-5 2022-01-06 [1] CRAN (R 4.2.3) cli 3.6.1 2023-03-23 [1] CRAN (R 4.2.3) colorspace 2.1-0 2023-01-23 [1] CRAN (R 4.2.3) crayon 1.5.2 2022-09-29 [1] CRAN (R 4.2.3) devtools 2.4.5 2022-10-11 [1] CRAN (R 4.2.3) digest 0.6.31 2022-12-11 [1] CRAN (R 4.2.3) dplyr 1.1.2 2023-04-20 [1] CRAN (R 4.2.3) ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.2.3) fansi 1.0.4 2023-01-22 [1] CRAN (R 4.2.3) fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.2.3) fs 1.6.2 2023-04-25 [1] CRAN (R 4.2.3) generics 0.1.3 2022-07-05 [1] CRAN (R 4.2.3) gginnards 0.1.1 2022-10-16 [1] CRAN (R 4.2.3) ggplot2 3.4.2 2023-04-03 [1] CRAN (R 4.2.3) ggpubr 0.6.0 2023-02-10 [1] CRAN (R 4.2.3) ggsignif 0.6.4 2022-10-13 [1] CRAN (R 4.2.3) glue 1.6.2 2022-02-24 [1] CRAN (R 4.2.3) gtable 0.3.3 2023-03-21 [1] CRAN (R 4.2.3) here 1.0.1 2020-12-13 [1] CRAN (R 4.2.3) htmltools 0.5.5 2023-03-23 [1] CRAN (R 4.2.3) htmlwidgets 1.6.2 2023-03-17 [1] CRAN (R 4.2.3) httpuv 1.6.9 2023-02-14 [1] CRAN (R 4.2.3) jsonlite 1.8.4 2022-12-06 [1] CRAN (R 4.2.3) later 1.3.0 2021-08-18 [1] CRAN (R 4.2.3) lattice 0.21-8 2023-04-05 [4] CRAN (R 4.2.3) lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.2.3) magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.2.3) Matrix 1.5-4 2023-04-04 [1] CRAN (R 4.2.3) memoise 2.0.1 2021-11-26 [1] CRAN (R 4.2.3) mime 0.12 2021-09-28 [1] CRAN (R 4.2.3) miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.2.3) munsell 0.5.0 2018-06-12 [1] CRAN (R 4.2.3) pillar 1.9.0 2023-03-22 [1] CRAN (R 4.2.3) pkgbuild 1.4.0 2022-11-27 [1] CRAN (R 4.2.3) pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.2.3) pkgload 1.3.2 2022-11-16 [1] CRAN (R 4.2.3) png 0.1-8 2022-11-29 [1] CRAN (R 4.2.3) prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.2.3) processx 3.8.1 2023-04-18 [1] CRAN (R 4.2.3) profvis 0.3.7 2020-11-02 [1] CRAN (R 4.2.3) promises 1.2.0.1 2021-02-11 [1] CRAN (R 4.2.3) ps 1.7.5 2023-04-18 [1] CRAN (R 4.2.3) purrr 1.0.1 2023-01-10 [1] CRAN (R 4.2.3) R6 2.5.1 2021-08-19 [1] CRAN (R 4.2.3) rappdirs 0.3.3 2021-01-31 [1] CRAN (R 4.2.3) Rcpp 1.0.10 2023-01-22 [1] CRAN (R 4.2.3) remotes 2.4.2 2021-11-30 [1] CRAN (R 4.2.3) reticulate 1.28 2023-01-27 [1] CRAN (R 4.2.3) rlang 1.1.0 2023-03-14 [1] CRAN (R 4.2.3) Rmagic 2.0.3 2019-11-21 [1] CRAN (R 4.2.3) rprojroot 2.0.3 2022-04-02 [1] CRAN (R 4.2.3) rstatix 0.7.2 2023-02-01 [1] CRAN (R 4.2.3) rstudioapi 0.14 2022-08-22 [1] CRAN (R 4.2.3) scales 1.2.1 2022-08-20 [1] CRAN (R 4.2.3) scATOMIC 1.1.4 2023-04-17 [1] Github (abelson-lab/scATOMIC@02dc6cb) sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.2.3) shiny 1.7.4 2022-12-15 [1] CRAN (R 4.2.3) stringi 1.7.12 2023-01-11 [1] CRAN (R 4.2.3) stringr 1.5.0 2022-12-02 [1] CRAN (R 4.2.3) tibble 3.2.1 2023-03-20 [1] CRAN (R 4.2.3) tidyr 1.3.0 2023-01-24 [1] CRAN (R 4.2.3) tidyselect 1.2.0 2022-10-10 [1] CRAN (R 4.2.3) urlchecker 1.0.1 2021-11-30 [1] CRAN (R 4.2.3) usethis 2.1.6 2022-05-25 [1] CRAN (R 4.2.3) utf8 1.2.3 2023-01-31 [1] CRAN (R 4.2.3) vctrs 0.6.2 2023-04-19 [1] CRAN (R 4.2.3) withr 2.5.0 2022-03-03 [1] CRAN (R 4.2.3) xtable 1.8-4 2019-04-21 [1] CRAN (R 4.2.3)

[1] /home/ubuntu/R/x86_64-pc-linux-gnu-library/4.2 [2] /usr/local/lib/R/site-library [3] /usr/lib/R/site-library [4] /usr/lib/R/library

inofechm commented 1 year ago

It looks like the cutoff package isnt loaded into your session, can you make sure it is installed. To install it run: devtools::install_github("choisy/cutoff")

This package is different from the CRAN cutoff package, please remove that package if you have it installed.

Let me know if this works!

u2sj commented 1 year ago

Thank you for your response. I have run the tutorial code after reinstalled cutoff package. it has been running for 1 hours and still hasn't produced any files.

─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────── setting value version R version 4.2.3 (2023-03-15) os Ubuntu 20.04.6 LTS system x86_64, linux-gnu ui RStudio language zh_CN:en collate zh_CN.UTF-8 ctype zh_CN.UTF-8 tz Asia/Shanghai date 2023-04-28 rstudio 2023.03.0+386 Cherry Blossom (desktop) pandoc NA

─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── package version date (UTC) lib source abind 1.4-5 2016-07-21 [1] CRAN (R 4.2.3) agrmt 1.42.8 2021-11-17 [1] CRAN (R 4.2.3) backports 1.4.1 2021-12-13 [1] CRAN (R 4.2.3) BiocManager 1.30.20 2023-02-24 [1] CRAN (R 4.2.3) broom 1.0.4 2023-03-11 [1] CRAN (R 4.2.3) cachem 1.0.7 2023-02-24 [1] CRAN (R 4.2.3) callr 3.7.3 2022-11-02 [1] CRAN (R 4.2.3) car 3.1-2 2023-03-30 [1] CRAN (R 4.2.3) carData 3.0-5 2022-01-06 [1] CRAN (R 4.2.3) caret 6.0-94 2023-03-21 [1] CRAN (R 4.2.3) class 7.3-21 2023-01-23 [4] CRAN (R 4.2.2) cli 3.6.1 2023-03-23 [1] CRAN (R 4.2.3) cluster 2.1.4 2022-08-22 [4] CRAN (R 4.2.1) codetools 0.2-19 2023-02-01 [4] CRAN (R 4.2.2) colorspace 2.1-0 2023-01-23 [1] CRAN (R 4.2.3) copykat 1.1.0 2023-04-17 [1] Github (navinlabcode/copykat@b795ff7) cowplot 1.1.1 2020-12-30 [1] CRAN (R 4.2.3) crayon 1.5.2 2022-09-29 [1] CRAN (R 4.2.3) cutoff 0.1.0 2023-04-28 [1] Github (choisy/cutoff@517686c) data.table 1.14.8 2023-02-17 [1] CRAN (R 4.2.3) deldir 1.0-6 2021-10-23 [1] CRAN (R 4.2.3) devtools 2.4.5 2022-10-11 [1] CRAN (R 4.2.3) digest 0.6.31 2022-12-11 [1] CRAN (R 4.2.3) dplyr 1.1.2 2023-04-20 [1] CRAN (R 4.2.3) ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.2.3) fansi 1.0.4 2023-01-22 [1] CRAN (R 4.2.3) fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.2.3) fitdistrplus 1.1-11 2023-04-25 [1] CRAN (R 4.2.3) foreach 1.5.2 2022-02-02 [1] CRAN (R 4.2.3) fs 1.6.2 2023-04-25 [1] CRAN (R 4.2.3) future 1.32.0 2023-03-07 [1] CRAN (R 4.2.3) future.apply 1.10.0 2022-11-05 [1] CRAN (R 4.2.3) generics 0.1.3 2022-07-05 [1] CRAN (R 4.2.3) gginnards 0.1.1 2022-10-16 [1] CRAN (R 4.2.3) ggplot2 3.4.2 2023-04-03 [1] CRAN (R 4.2.3) ggpubr 0.6.0 2023-02-10 [1] CRAN (R 4.2.3) ggrepel 0.9.3 2023-02-03 [1] CRAN (R 4.2.3) ggridges 0.5.4 2022-09-26 [1] CRAN (R 4.2.3) ggsignif 0.6.4 2022-10-13 [1] CRAN (R 4.2.3) globals 0.16.2 2022-11-21 [1] CRAN (R 4.2.3) glue 1.6.2 2022-02-24 [1] CRAN (R 4.2.3) goftest 1.2-3 2021-10-07 [1] CRAN (R 4.2.3) gower 1.0.1 2022-12-22 [1] CRAN (R 4.2.3) gridExtra 2.3 2017-09-09 [1] CRAN (R 4.2.3) gtable 0.3.3 2023-03-21 [1] CRAN (R 4.2.3) hardhat 1.3.0 2023-03-30 [1] CRAN (R 4.2.3) here 1.0.1 2020-12-13 [1] CRAN (R 4.2.3) htmltools 0.5.5 2023-03-23 [1] CRAN (R 4.2.3) htmlwidgets 1.6.2 2023-03-17 [1] CRAN (R 4.2.3) httpuv 1.6.9 2023-02-14 [1] CRAN (R 4.2.3) httr 1.4.5 2023-02-24 [1] CRAN (R 4.2.3) ica 1.0-3 2022-07-08 [1] CRAN (R 4.2.3) igraph 1.4.2 2023-04-07 [1] CRAN (R 4.2.3) ipred 0.9-14 2023-03-09 [1] CRAN (R 4.2.3) irlba 2.3.5.1 2022-10-03 [1] CRAN (R 4.2.3) iterators 1.0.14 2022-02-05 [1] CRAN (R 4.2.3) jsonlite 1.8.4 2022-12-06 [1] CRAN (R 4.2.3) KernSmooth 2.23-20 2021-05-03 [1] CRAN (R 4.2.3) later 1.3.0 2021-08-18 [1] CRAN (R 4.2.3) lattice 0.21-8 2023-04-05 [4] CRAN (R 4.2.3) lava 1.7.2.1 2023-02-27 [1] CRAN (R 4.2.3) lazyeval 0.2.2 2019-03-15 [1] CRAN (R 4.2.3) leiden 0.4.3 2022-09-10 [1] CRAN (R 4.2.3) lifecycle 1.0.3 2022-10-07 [1] CRAN (R 4.2.3) listenv 0.9.0 2022-12-16 [1] CRAN (R 4.2.3) lmtest 0.9-40 2022-03-21 [1] CRAN (R 4.2.3) lubridate 1.9.2 2023-02-10 [1] CRAN (R 4.2.3) magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.2.3) MASS 7.3-59 2023-04-21 [4] CRAN (R 4.2.3) Matrix 1.5-4 2023-04-04 [1] CRAN (R 4.2.3) matrixStats 0.63.0 2022-11-18 [1] CRAN (R 4.2.3) memoise 2.0.1 2021-11-26 [1] CRAN (R 4.2.3) mime 0.12 2021-09-28 [1] CRAN (R 4.2.3) miniUI 0.1.1.1 2018-05-18 [1] CRAN (R 4.2.3) ModelMetrics 1.2.2.2 2020-03-17 [1] CRAN (R 4.2.3) munsell 0.5.0 2018-06-12 [1] CRAN (R 4.2.3) nlme 3.1-162 2023-01-31 [4] CRAN (R 4.2.2) nnet 7.3-18 2022-09-28 [4] CRAN (R 4.2.1) parallelly 1.35.0 2023-03-23 [1] CRAN (R 4.2.3) patchwork 1.1.2 2022-08-19 [1] CRAN (R 4.2.3) pbapply 1.7-0 2023-01-13 [1] CRAN (R 4.2.3) pillar 1.9.0 2023-03-22 [1] CRAN (R 4.2.3) pkgbuild 1.4.0 2022-11-27 [1] CRAN (R 4.2.3) pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.2.3) pkgload 1.3.2 2022-11-16 [1] CRAN (R 4.2.3) plotly 4.10.1 2022-11-07 [1] CRAN (R 4.2.3) plyr 1.8.8 2022-11-11 [1] CRAN (R 4.2.3) png 0.1-8 2022-11-29 [1] CRAN (R 4.2.3) polyclip 1.10-4 2022-10-20 [1] CRAN (R 4.2.3) prettyunits 1.1.1 2020-01-24 [1] CRAN (R 4.2.3) pROC 1.18.0 2021-09-03 [1] CRAN (R 4.2.3) processx 3.8.1 2023-04-18 [1] CRAN (R 4.2.3) prodlim 2023.03.31 2023-04-02 [1] CRAN (R 4.2.3) profvis 0.3.7 2020-11-02 [1] CRAN (R 4.2.3) progressr 0.13.0 2023-01-10 [1] CRAN (R 4.2.3) promises 1.2.0.1 2021-02-11 [1] CRAN (R 4.2.3) ps 1.7.5 2023-04-18 [1] CRAN (R 4.2.3) purrr 1.0.1 2023-01-10 [1] CRAN (R 4.2.3) R6 2.5.1 2021-08-19 [1] CRAN (R 4.2.3) randomForest 4.7-1.1 2022-05-23 [1] CRAN (R 4.2.3) RANN 2.6.1 2019-01-08 [1] CRAN (R 4.2.3) rappdirs 0.3.3 2021-01-31 [1] CRAN (R 4.2.3) RColorBrewer 1.1-3 2022-04-03 [1] CRAN (R 4.2.3) Rcpp 1.0.10 2023-01-22 [1] CRAN (R 4.2.3) RcppAnnoy 0.0.20 2022-10-27 [1] CRAN (R 4.2.3) recipes 1.0.6 2023-04-25 [1] CRAN (R 4.2.3) remotes 2.4.2 2021-11-30 [1] CRAN (R 4.2.3) reshape2 1.4.4 2020-04-09 [1] CRAN (R 4.2.3) reticulate 1.28 2023-01-27 [1] CRAN (R 4.2.3) rlang 1.1.0 2023-03-14 [1] CRAN (R 4.2.3) Rmagic 2.0.3 2019-11-21 [1] CRAN (R 4.2.3) ROCR 1.0-11 2020-05-02 [1] CRAN (R 4.2.3) rpart 4.1.19 2022-10-21 [4] CRAN (R 4.2.1) rprojroot 2.0.3 2022-04-02 [1] CRAN (R 4.2.3) rstatix 0.7.2 2023-02-01 [1] CRAN (R 4.2.3) rstudioapi 0.14 2022-08-22 [1] CRAN (R 4.2.3) Rtsne 0.16 2022-04-17 [1] CRAN (R 4.2.3) scales 1.2.1 2022-08-20 [1] CRAN (R 4.2.3) scATOMIC 1.1.4 2023-04-17 [1] Github (abelson-lab/scATOMIC@02dc6cb) scattermore 0.8 2022-02-14 [1] CRAN (R 4.2.3) sctransform 0.3.5 2022-09-21 [1] CRAN (R 4.2.3) sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.2.3) Seurat 4.3.0 2022-11-18 [1] CRAN (R 4.2.3) SeuratObject * 4.1.3 2022-11-07 [1] CRAN (R 4.2.3) shiny 1.7.4 2022-12-15 [1] CRAN (R 4.2.3) sp 1.6-0 2023-01-19 [1] CRAN (R 4.2.3) spatstat.data 3.0-1 2023-03-12 [1] CRAN (R 4.2.3) spatstat.explore 3.1-0 2023-03-14 [1] CRAN (R 4.2.3) spatstat.geom 3.1-0 2023-03-12 [1] CRAN (R 4.2.3) spatstat.random 3.1-4 2023-03-13 [1] CRAN (R 4.2.3) spatstat.sparse 3.0-1 2023-03-12 [1] CRAN (R 4.2.3) spatstat.utils 3.0-2 2023-03-11 [1] CRAN (R 4.2.3) stringi 1.7.12 2023-01-11 [1] CRAN (R 4.2.3) stringr 1.5.0 2022-12-02 [1] CRAN (R 4.2.3) survival 3.5-5 2023-03-12 [4] CRAN (R 4.2.3) tensor 1.5 2012-05-05 [1] CRAN (R 4.2.3) tibble 3.2.1 2023-03-20 [1] CRAN (R 4.2.3) tidyr 1.3.0 2023-01-24 [1] CRAN (R 4.2.3) tidyselect 1.2.0 2022-10-10 [1] CRAN (R 4.2.3) timechange 0.2.0 2023-01-11 [1] CRAN (R 4.2.3) timeDate 4022.108 2023-01-07 [1] CRAN (R 4.2.3) urlchecker 1.0.1 2021-11-30 [1] CRAN (R 4.2.3) usethis 2.1.6 2022-05-25 [1] CRAN (R 4.2.3) utf8 1.2.3 2023-01-31 [1] CRAN (R 4.2.3) uwot 0.1.14 2022-08-22 [1] CRAN (R 4.2.3) vctrs 0.6.2 2023-04-19 [1] CRAN (R 4.2.3) viridisLite 0.4.1 2022-08-22 [1] CRAN (R 4.2.3) withr 2.5.0 2022-03-03 [1] CRAN (R 4.2.3) xtable 1.8-4 2019-04-21 [1] CRAN (R 4.2.3) zoo 1.8-12 2023-04-13 [1] CRAN (R 4.2.3)

[1] /home/ubuntu/R/x86_64-pc-linux-gnu-library/4.2 [2] /usr/local/lib/R/site-library [3] /usr/lib/R/site-library [4] /usr/lib/R/library

─ Python configuration ─────────────────────────────────────────────────────────────────────────────────────────────────────────── python: /home/ubuntu/anaconda3/envs/r-reticulate/bin/python libpython: /home/ubuntu/anaconda3/envs/r-reticulate/lib/libpython3.9.so pythonhome: /home/ubuntu/anaconda3/envs/r-reticulate:/home/ubuntu/anaconda3/envs/r-reticulate version: 3.9.16 | packaged by conda-forge | (main, Feb 1 2023, 21:39:03) [GCC 11.3.0] numpy: /home/ubuntu/anaconda3/envs/r-reticulate/lib/python3.9/site-packages/numpy numpy_version: 1.24.3 magic: /home/ubuntu/anaconda3/envs/r-reticulate/lib/python3.9/site-packages/magic

NOTE: Python version was forced by use_python function

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

inofechm commented 1 year ago

I'm sorry about this issue, i've never had an issue running it on ubuntu so I am a bit stumped. How many cores does your machine have, I am thinking maybe too many parallel processes are being used. Can you try two other things:

can you please run the run_scATOMIC function with mc.cores=1 ?

can you run library(Rmagic) data(magic_testdata) data_MAGIC <- magic(magic_testdata) and show the output?

inofechm commented 1 year ago

can you also try cell_predictions <- run_scATOMIC(lung_cancer_demo_data, confidence_cutoff= F ,mc.cores = 1) theres a chance that for some reason on your end the confidence cutoff isnt working properly.

inofechm commented 1 year ago

ive also modified the cutoff process in v2 (released today), so please try that as well. You can update scATOMIC by re-installing it with devtools.

u2sj commented 1 year ago

library(Rmagic) 载入需要的程辑包：Matrix data(magic_testdata) data_MAGIC <- magic(magic_testdata) Calculating MAGIC... Running MAGIC on 500 cells and 197 genes. /home/ubuntu/anaconda3/envs/r-reticulate/lib/python3.9/site-packages/magic/magic.py:425: UserWarning: Input matrix contains unexpressed genes. Please remove them prior to running MAGIC. warnings.warn( Calculating graph and diffusion operator... Calculating PCA... there is no result was showed for for 30 minutes running.

And I found that : library(reticulate)

use_condaenv(conda = "auto", condaenv = "r-reticulate") install.magic() Attempting to install MAGIC python package with reticulate Using virtual environment 'r-reticulate' ... Error in value[3L] : Cannot locate MAGIC Python package, please install through pip (e.g. pip install magic-impute) and then restart R. 此外: Warning message: In system2(python, c("-c", shQuote(command)), stdout = TRUE, stderr = TRUE) : 运行命令''/home/ubuntu/.virtualenvs/r-reticulate/bin/python' -c 'import sys; import pip; sys.stdout.write(pip.version)' 2>&1'的状态是1 pymagic_is_available() [1] TRUE

Could you help me find out what is wrong with this？ Thanks！

u2sj commented 1 year ago

lung_cancer_demo_data <- demo_lung_data

Preprocessing dataset

pct_mt <- colSums(lung_cancer_demo_data[grep("^MT-", row.names(lung_cancer_demo_data)),])/colSums(lung_cancer_demo_data) * 100 nFeatureRNA <- colSums(lung_cancer_demo_data > 0) lung_cancer_demo_data <- lung_cancer_demo_data[, names(which(pct_mt < 15))] lung_cancer_demo_data <- lung_cancer_demo_data[, intersect(names(which(nFeatureRNA > 500)), colnames(lung_cancer_demo_data))]

Running scATOMIC

cell_predictions <- run_scATOMIC(lung_cancer_demo_data, confidence_cutoff= F ,mc.cores = (parallel::detectCores() - 1))

Even though I installed the v2 package, it doesn't seem to be working properly with cell_predictions <- run_scATOMIC(lung_cancer_demo_data, confidence_cutoff= F ,mc.cores = 1).

inofechm commented 1 year ago

given the error:

Attempting to install MAGIC python package with reticulate Using virtual environment 'r-reticulate' ... Error in value[3L] : Cannot locate MAGIC Python package, please install through pip (e.g. pip install magic-impute) and then restart R. 此外: Warning message: In system2(python, c("-c", shQuote(command)), stdout = TRUE, stderr = TRUE) : 运行命令''/home/ubuntu/.virtualenvs/r-reticulate/bin/python' -c 'import sys; import pip; sys.stdout.write(pip.version)'

It seems like your R reticulate isnt interfacing with python and MAGIC properly. I believe the python reticulate is using is missing the MAGIC installation.

Have you installed the python implementation of MAGIC in the venv being used?, it seems like the wrong python environment is being used.

can you run reticulate::py_config() and show the output?

I think you need to activate the r-reticulate venv that is being used by R in the command line and then run pip install magic-impute

you can then change the environment R is using by restarting R and one of the following depending how you manage your python envs: library(reticulate) reticulate::use_python() reticulate::use_condaenv() reticulate::use_virtualenv()

following this restart R and try this again: library(Rmagic) data(magic_testdata) data_MAGIC <- magic(magic_testdata)

u2sj commented 1 year ago

reticulate::py_config() python: /home/ubuntu/anaconda3/envs/r-reticulate/bin/python3.9 libpython: /home/ubuntu/anaconda3/envs/r-reticulate/lib/libpython3.9.so pythonhome: /home/ubuntu/anaconda3/envs/r-reticulate:/home/ubuntu/anaconda3/envs/r-reticulate version: 3.9.16 | packaged by conda-forge | (main, Feb 1 2023, 21:39:03) [GCC 11.3.0] numpy: /home/ubuntu/anaconda3/envs/r-reticulate/lib/python3.9/site-packages/numpy numpy_version: 1.24.3 magic: /home/ubuntu/anaconda3/envs/r-reticulate/lib/python3.9/site-packages/magic

I have try to install the magic-impute in the r-reticulate env, and when I run the codes as followed: library(Rmagic) data(magic_testdata) data_MAGIC <- magic(magic_testdata)

it showed that: [41371:41371:20230503,213329.646974:ERROR elf_dynamic_array_reader.h:61] tag not found [41371:41372:20230503,213329.676504:ERROR directory_reader_posix.cc:42] opendir: No such file or directory (2) and R session aborted.

inofechm commented 1 year ago

Thanks for the helpful troubleshooting, It looks like you are having an error that has been previously associated with RStudio on Ubuntu 20.04.

There is a chance you just need to update your drivers, see the following issues: https://forums.autodesk.com/t5/eagle-forum/ubuntu-20-04-stopped-working-with-eagle/m-p/9713956#M31116 https://github.com/rstudio/rstudio/issues/7310 https://github.com/TheoreticalEcology/s-jSDM/issues/102 https://community.rstudio.com/t/r-studio-crashes-with-fatal-error-elf-dynamic-array-reader-h-61-tag-not-found/93130

We previously tested everything on Ubuntu 18, so I can work on checking if MAGIC works on Ubuntu 20.04.

inofechm commented 1 year ago

Can you check if you can run the magic diagnostic in the interactive mode of R in command line? just open a terminal/command line and type R and you should be in an interactive session. Please let me know MAGIC works there.

u2sj commented 1 year ago

library(Rmagic) data(magic_testdata) data_MAGIC <- magic(magic_testdata) 载入需要的程辑包：Matrix Calculating MAGIC... Running MAGIC on 500 cells and 197 genes. /home/ubuntu/anaconda3/envs/r-reticulate/lib/python3.9/site-packages/magic/magic.py:425: UserWarning: Input matrix contains unexpressed genes. Please remove them prior to running MAGIC. warnings.warn( Calculating graph and diffusion operator... Calculating PCA... Calculated PCA in 0.90 seconds. Calculating KNN search... /home/ubuntu/anaconda3/envs/r-reticulate/lib/python3.9/site-packages/graphtools/graphs.py:290: RuntimeWarning: Detected zero distance between 10 pairs of samples. Consider removing duplicates to avoid errors in downstream processing. warnings.warn( Calculated KNN search in 0.04 seconds. Calculating affinities... Calculated affinities in 0.05 seconds. Calculated graph and diffusion operator in 1.00 seconds. Calculating imputation... Calculated MAGIC in 1.00 seconds.

It seem to be run properly in command line. And my computer doesn't have a graphics card，could I still use this package in R studio? thanks!

inofechm commented 1 year ago

Yay I'm glad we found the issue! Unfortunately, since this issue has to do with MAGIC, I cannot change their code base. What you can do though is run the scATOMIC pipeline in command line, then save the output object from create_summary_matrix() as an RDS file using: saveRDS(results_matrix, "PATH_SAVE_FILE.RDS") then in Rstudio just run results_matrix <- readRDS("PATH_SAVE_FILE.RDS") and you will be able to load the file and continue with anything you want to do in Rstudio

I am closing the issue for now, feel free to comment if you have any other questions! Cheers, Ido