Error in seurat to anndata

[massonix]

Dear cellgeni team,

I'm trying to convert the pbmc_small Seurat object that is loaded by Seurat by default to anndata with sceasy. I use the following code:

library(Seurat)
library(reticulate)
library(sceasy)
use_condaenv("sceasy", required = TRUE)
sceasy::convertFormat(
    pbmc_small, 
    from = "seurat", 
    to = "anndata", 
    outFile = "pbmc_small.h5ad"
)

However, I get the following error:

Error in py_module_import(module, convert = convert) : 
  SyntaxError: invalid syntax (base.py, line 66)

Detailed traceback: 
  File "/anaconda3/envs/sceasy/lib/python2.7/site-packages/anndata/__init__.py", line 1, in <module>
    from .base import AnnData, _MAIN_NARRATIVE
In addition: Warning message:
In .regularise_df(obj@meta.data) :
  Dropping single category variables:orig.ident

Here is my R session

R version 3.6.2 (2019-12-12)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: macOS High Sierra 10.13.6

Matrix products: default
BLAS/LAPACK: /anaconda3/envs/sceasy/lib/libopenblasp-r0.3.7.dylib

locale:
[1] C/UTF-8/C/C/C/C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] sceasy_0.0.1    reticulate_1.14 Seurat_3.0.2   

loaded via a namespace (and not attached):
 [1] httr_1.4.1          tidyr_1.0.0         viridisLite_0.3.0  
 [4] jsonlite_1.6        splines_3.6.2       lsei_1.2-0         
 [7] R.utils_2.9.2       gtools_3.8.1        Rdpack_0.11-1      
[10] assertthat_0.2.1    ggrepel_0.8.1       globals_0.12.5     
[13] backports_1.1.5     pillar_1.4.3        lattice_0.20-38    
[16] glue_1.3.1          digest_0.6.23       RColorBrewer_1.1-2 
[19] SDMTools_1.1-221.2  colorspace_1.4-1    cowplot_1.0.0      
[22] htmltools_0.4.0     Matrix_1.2-18       R.oo_1.23.0        
[25] plyr_1.8.5          pkgconfig_2.0.3     bibtex_0.4.2.2     
[28] tsne_0.1-3          listenv_0.8.0       purrr_0.3.3        
[31] scales_1.1.0        RANN_2.6.1          gdata_2.18.0       
[34] Rtsne_0.15          tibble_2.1.3        ggplot2_3.2.1      
[37] ROCR_1.0-7          pbapply_1.4-2       lazyeval_0.2.2     
[40] survival_3.1-8      magrittr_1.5        crayon_1.3.4       
[43] R.methodsS3_1.7.1   future_1.15.1       nlme_3.1-143       
[46] MASS_7.3-51.5       gplots_3.0.1.1      ica_1.0-2          
[49] tools_3.6.2         fitdistrplus_1.0-14 data.table_1.12.8  
[52] gbRd_0.4-11         lifecycle_0.1.0     stringr_1.4.0      
[55] plotly_4.9.1        munsell_0.5.0       cluster_2.1.0      
[58] irlba_2.3.3         compiler_3.6.2      rsvd_1.0.2         
[61] caTools_1.17.1.3    rlang_0.4.2         grid_3.6.2         
[64] ggridges_0.5.1      htmlwidgets_1.5.1   igraph_1.2.4.2     
[67] bitops_1.0-6        npsurv_0.4-0        gtable_0.3.0       
[70] codetools_0.2-16    reshape2_1.4.3      R6_2.4.1           
[73] gridExtra_2.3       zoo_1.8-6           dplyr_0.8.3        
[76] zeallot_0.1.0       future.apply_1.4.0  KernSmooth_2.23-16 
[79] metap_1.1           ape_5.3             stringi_1.4.3      
[82] parallel_3.6.2      Rcpp_1.0.3          sctransform_0.2.1  
[85] png_0.1-7           vctrs_0.2.1         tidyselect_0.2.5   
[88] lmtest_0.9-37    

If you could shed some light it'd be extremely useful.

Thanks a lot,

Ramon

[nh3]

Hi @massonix, thanks for the feedback. It appears the version of python in your environment is 2.7. For seurat to anndata conversion, it requires an environment with python package anndata (which depends on python3) installed. Please follow the installation section of readme.

[massonix]

Hi,

Thanks for your response.

I am still getting some errors, I will try to be more detailed with the code. This is how I set the environment:

conda create -n cellgeni4
conda activate cellgeni4
conda install -c bioconda r-sceasy
conda install anndata==0.6.19 -c bioconda
conda install loompy==2.0.17 -c bioconda

Anndata and loompy are correctly installed:

python
Python 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 02:16:08) 
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import anndata
>>> import loompy
>>> 

In R:

pbmc <- readRDS("~/Desktop/pbmc_small.rds")
.libPaths()
[1] "/opt/anaconda3/envs/cellgeni4/lib/R/library"
library(reticulate)
use_condaenv('cellgeni4')
loompy <- reticulate::import('loompy')
Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'loompy'
sceasy::convertFormat(pbmc, from="seurat", to="anndata",outFile ="~/Desktop/pbmc.h5ad")
Error in py_module_import(module, convert = convert) : 
  ModuleNotFoundError: No module named 'anndata'
In addition: Warning message:
In .regularise_df(obj@meta.data) :
  Dropping single category variables:orig.ident

I tried to change the import path of reticulate:

reticulate_path <- "/opt/anaconda3/envs/cellgeni4/lib/python3.6/site-packages/"
Sys.setenv(RETICULATE_PYTHON = reticulate_path)
loompy <- reticulate::import('loompy')
Error in system2(command = python, args = paste0("\"", config_script,  : 
  error in running command

But it is still not working.

Thanks again for your precious help. This package is what our single-cell community needs right now.

R

[nh3]

The error is apparently caused by reticulate failing to find loompy and anndata. Worth checking what reticulate::py_discover_config() returns.

[massonix]

This is what it returns:

reticulate::py_discover_config()
Error in system2(command = python, args = paste0("\"", config_script,  : 
  error in running command

I will explore it further

[massonix]

It seems that I had to specify the full path to the environment:

reticulate::use_condaenv("/opt/anaconda3/envs/cellgeni4/")

and with that it worked fine.

Thanks for the help!

R

[rschiavo16]

Hi there! I have been having a similar issue. I am having trouble getting reticulate to recognize my conda enviornment. Everytime I try to run the convert code I get: Error in py_module_import(module, convert = convert) : ModuleNotFoundError: No module named 'anndata' .
I ran reticulate::py_discover_config() and it returned this so I don't think I'm having the same path issue as above. Any insight would be greatly appreciated!

python: C:/Users/bexx1/Miniconda3/envs/chill/python.exe libpython: C:/Users/bexx1/Miniconda3/envs/chill/python37.dll pythonhome: C:/Users/bexx1/Miniconda3/envs/chill version: 3.7.6 (default, Jan 8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)] Architecture: 64bit numpy: C:/Users/bexx1/Miniconda3/envs/chill/Lib/site-packages/numpy numpy_version: 1.18.1

python versions found: C:/Users/bexx1/Miniconda3/envs/chill/python.exe C:/Users/bexx1/Anaconda3/envs/r-reticulate/python.exe C:/Users/bexx1/Anaconda3/python.exe C:/Users/bexx1/AppData/Local/Programs/Python/Python37-32/python.exe C:/Users/bexx1/Miniconda3/python.exe C:/Users/bexx1/Miniconda3/envs/cold/python.exe C:/Users/bexx1/Miniconda3/envs/hello/python.exe C:/Users/bexx1/Miniconda3/envs/snow/python.exe

Here is my R session Info

R version 3.5.3 (2019-03-11) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 18362)

Matrix products: default

locale: [1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] sceasy_0.0.5 reticulate_1.14

loaded via a namespace (and not attached): [1] Seurat_3.1.2 TH.data_1.0-10 Rtsne_0.15 colorspace_1.4-1 ellipsis_0.3.0
[6] ggridges_0.5.2 rprojroot_1.3-2 fs_1.3.1 rstudioapi_0.10 leiden_0.3.3
[11] listenv_0.8.0 npsurv_0.4-0 remotes_2.1.0 ggrepel_0.8.1 bit64_0.9-7
[16] fansi_0.4.1 mvtnorm_1.0-12 codetools_0.2-16 splines_3.5.3 R.methodsS3_1.7.1
[21] mnormt_1.5-6 lsei_1.2-0 TFisher_0.2.0 pkgload_1.0.2 jsonlite_1.6.1
[26] ica_1.0-2 cluster_2.0.7-1 png_0.1-7 R.oo_1.23.0 uwot_0.1.5
[31] sctransform_0.2.1 BiocManager_1.30.10 compiler_3.5.3 httr_1.4.1 backports_1.1.5
[36] assertthat_0.2.1 Matrix_1.2-15 lazyeval_0.2.2 cli_2.0.1 prettyunits_1.1.1
[41] htmltools_0.4.0 tools_3.5.3 rsvd_1.0.2 igraph_1.2.4.2 gtable_0.3.0
[46] glue_1.3.1 RANN_2.6.1 reshape2_1.4.3 dplyr_0.8.4 rappdirs_0.3.1
[51] Rcpp_1.0.3 Biobase_2.42.0 vctrs_0.2.2 multtest_2.38.0 gdata_2.18.0
[56] ape_5.3 nlme_3.1-137 gbRd_0.4-11 lmtest_0.9-37 loomR_0.2.1.9000
[61] stringr_1.4.0 ps_1.3.0 globals_0.12.5 testthat_2.3.1 lifecycle_0.1.0
[66] irlba_2.3.3 devtools_2.2.1 gtools_3.8.1 future_1.16.0 MASS_7.3-51.1
[71] zoo_1.8-7 scales_1.1.0 parallel_3.5.3 sandwich_2.5-1 RColorBrewer_1.1-2 [76] memoise_1.1.0 pbapply_1.4-2 gridExtra_2.3 ggplot2_3.2.1 stringi_1.4.5
[81] desc_1.2.0 mutoss_0.1-12 plotrix_3.7-7 caTools_1.17.1.3 BiocGenerics_0.28.0 [86] pkgbuild_1.0.6 bibtex_0.4.2.2 Rdpack_0.11-1 SDMTools_1.1-221.2 rlang_0.4.4
[91] pkgconfig_2.0.3 bitops_1.0-6 lattice_0.20-38 ROCR_1.0-7 purrr_0.3.3
[96] htmlwidgets_1.5.1 processx_3.4.1 cowplot_1.0.0 bit_1.1-15.1 tidyselect_1.0.0
[101] RcppAnnoy_0.0.14 plyr_1.8.5 magrittr_1.5 R6_2.4.1 gplots_3.0.1.2
[106] multcomp_1.4-12 withr_2.1.2 pillar_1.4.3 sn_1.5-5 fitdistrplus_1.0-14 [111] survival_2.43-3 tsne_0.1-3 tibble_2.1.3 future.apply_1.4.0 crayon_1.3.4
[116] hdf5r_1.3.1.9000 KernSmooth_2.23-15 plotly_4.9.1 usethis_1.5.1 grid_3.5.3
[121] data.table_1.12.8 callr_3.4.1 metap_1.3 digest_0.6.23 tidyr_1.0.2
[126] numDeriv_2016.8-1.1 R.utils_2.9.2 RcppParallel_4.4.4 stats4_3.5.3 munsell_0.5.0

[massonix]

Hi @rschiavo16 ! Quick question, what do you get if you run:

conda activate chill
python
import anndata

This will help narrow down your issue

[massonix]

@rschiavo16 You can download the .yaml file to recreate the conda environment that worked for me here

Hope you find it useful :)

R

[nh3]

@rschiavo16, we haven't tested in a Windows environment and I am not familiar with the peculiarities of conda/R installations in windows. Common things to check include whether you are able to import anndata in your conda environment outside R as @massonix suggested.

[rschiavo16]

Hi guys, thank you for your help with this! I was able to import anndata but while importing loompy into my conda environment I get

import anndata import loompy Traceback (most recent call last): File "", line 1, in File "C:\Users\bexx1\Miniconda3\envs\snow\lib\site-packages\loompy__init__.py", line 1, in from .utils import * File "C:\Users\bexx1\Miniconda3\envs\snow\lib\site-packages\loompy\utils.py", line 9 logging.warn(f"\u256d── " + message) ^ SyntaxError: invalid syntax

After returning to R to see if it was working over there I get

reticulate::py_discover_config() python: C:/Users/bexx1/Miniconda3/envs/snow/python.exe libpython: C:/Users/bexx1/Miniconda3/envs/snow/python35.dll pythonhome: C:/Users/bexx1/Miniconda3/envs/snow version: 3.5.6 |Anaconda, Inc.| (default, Aug 26 2018, 16:05:27) [MSC v.1900 64 bit (AMD64)] Architecture: 64bit numpy: C:/Users/bexx1/Miniconda3/envs/snow/Lib/site-packages/numpy numpy_version: 1.14.2

python versions found: C:/Users/bexx1/Miniconda3/envs/snow/python.exe C:/Users/bexx1/Anaconda3/envs/r-reticulate/python.exe C:/Users/bexx1/Anaconda3/python.exe C:/Users/bexx1/AppData/Local/Programs/Python/Python37-32/python.exe C:/Users/bexx1/Miniconda3/python.exe

reticulate::use_condaenv('snow') loompy <- reticulate::import('loompy') Error in py_module_import(module, convert = convert) : ModuleNotFoundError: No module named 'loompy' sceasy::convertFormat(alldata, from="seurat", to="anndata", outFile='filename.h5ad') Error in py_module_import(module, convert = convert) : ModuleNotFoundError: No module named 'anndata'

[nh3]

The error of importing loompy was due to your python version cannot cope with python >=3.6 syntax. The modules themselves were found. When importing in reticulate, the modules cannot be found at all. I suspect reticulate::use_condaenv('snow') is not working as intended. It's perhaps worth trying reticulate::use_condaenv('C:/Users/bexx1/Miniconda3/envs/snow') as that worked for @massonix.

[massonix]

Try also:

reticulate::use_virtualenv('C:/Users/bexx1/Miniconda3/envs/snow')

[rschiavo16]

I tried both of your suggestions and am still getting the same error :/ Potentially a Microsoft compatibility issue? Thank you so much for taking the time to help me with this!

[christensensm]

@rschiavo16 @massonix did you figure out a fix for this? Having the same issue on a mac. I can import anndata and loompy into python in my environment w/o errors but getting the same error in R. Is it a version problem with Python? Python 3.7.6 is my default

reticulate::py_discover_config() python: /home/chriss41/miniconda3/envs/myenv/bin/python libpython: /home/chriss41/miniconda3/envs/myenv/lib/libpython3.7m.so pythonhome: /home/chriss41/miniconda3/envs/myenv:/home/chriss41/miniconda3/envs/myenv version: 3.7.6 (default, Jan 8 2020, 19:59:22) [GCC 7.3.0] numpy: /home/chriss41/miniconda3/envs/myenv/lib/python3.7/site-packages/numpy numpy_version: 1.18.1

python versions found: /home/chriss41/miniconda3/envs/myenv/bin/python /home/chriss41/miniconda3/bin/python3 /usr/bin/python /home/chriss41/miniconda3/bin/python

[macrinalobo]

The solution that worked for me was setting the reticulate python version in the environment variable before loading the reticulate library in R:

Sys.setenv(RETICULATE_PYTHON="/Users//miniconda3/envs/sc_cellgeni_0813/bin/python3") library(reticulate) library(sceasy)

then:

reticulate::py_discover_config() python: /Users//miniconda3/envs/sc_cellgeni_0813/bin/python3 libpython: /Users//miniconda3/envs/sc_cellgeni_0813/lib/libpython3.7m.dylib pythonhome: /Users//miniconda3/envs/sc_cellgeni_0813:/Users/LoboM/miniconda3/envs/sc_cellgeni_0813 version: 3.7.4 (default, Aug 13 2019, 15:17:50) [Clang 4.0.1 (tags/RELEASE_401/final)] numpy: /Users//miniconda3/envs/sc_cellgeni_0813/lib/python3.7/site-packages/numpy numpy_version: 1.19.1

[bsierieb1]

the solution of @macrinalobo worked for me, too, thanks!

[sscien]

I end up fixing this by pip install loompy