search

drieslab / Giotto

Spatial omics analysis toolbox
https://drieslab.github.io/Giotto_website/
Other
258 stars 98 forks source link

problem with Giotto python environment #139

Closed orrzor closed 2 years ago

orrzor commented 3 years ago

Hello, I am having an issue getting the Giotto (v1.0.4) python environment working while running on a linux environment. I have tried using both installGiottoEnvironment() and creating the conda environment myself. In the two examples below, I get the message that python packages are missing, even though they have been installed in the appropriate conda environments. Notably, I do not get this missing package issue when running on my local Mac OSX machine. Any help would be appreciated. Thank you! -Orr

  1. Letting Giotto create the environment. 
    
library(Giotto)
library(data.table)
library(Matrix)
installGiottoEnvironment()
bead_positions <- fread(file="/broad/regevtmp/orr/slideseq/BeadLocationsForR.csv")
expr_matrix<-fread(file="/broad/regevtmp/orr/slideseq/MappedDGEForR.csv")
expr_mat <- as.matrix(expr_matrix[,-1]);rownames(expr_mat) = expr_matrix$Row
Slide_test <- createGiottoObject(raw_exprs = expr_mat, spatial_locs = bead_positions[,.(xcoord, ycoord)])

Package Plan

environment location: /home/unix/orr/.local/share/r-miniconda/envs/giotto_env

added / updated specs:

The following NEW packages will be INSTALLED:

arpack conda-forge/linux-64::arpack-3.7.0-hdefa2d7_2 cycler conda-forge/noarch::cycler-0.10.0-py_2 decorator conda-forge/noarch::decorator-4.4.2-py_0 freetype conda-forge/linux-64::freetype-2.10.4-h0708190_1 glpk conda-forge/linux-64::glpk-4.65-h9202a9a_1004 gmp conda-forge/linux-64::gmp-6.2.1-h58526e2_0 icu conda-forge/linux-64::icu-68.1-h58526e2_0 igraph conda-forge/linux-64::igraph-0.9.4-ha184e22_0 jbig conda-forge/linux-64::jbig-2.1-h7f98852_2003 joblib conda-forge/noarch::joblib-1.0.1-pyhd8ed1ab_0 jpeg conda-forge/linux-64::jpeg-9d-h36c2ea0_0 kiwisolver pkgs/main/linux-64::kiwisolver-1.3.1-py36h2531618_0 lcms2 conda-forge/linux-64::lcms2-2.12-hddcbb42_0 leidenalg conda-forge/linux-64::leidenalg-0.8.7-py36hc4f0c31_0 lerc conda-forge/linux-64::lerc-2.2.1-h9c3ff4c_0 libblas conda-forge/linux-64::libblas-3.9.0-11_linux64_openblas libcblas conda-forge/linux-64::libcblas-3.9.0-11_linux64_openblas libdeflate conda-forge/linux-64::libdeflate-1.7-h7f98852_5 libgfortran-ng conda-forge/linux-64::libgfortran-ng-11.2.0-h69a702a_9 libgfortran5 conda-forge/linux-64::libgfortran5-11.2.0-h5c6108e_9 libiconv conda-forge/linux-64::libiconv-1.16-h516909a_0 liblapack conda-forge/linux-64::liblapack-3.9.0-11_linux64_openblas libopenblas conda-forge/linux-64::libopenblas-0.3.17-pthreads_h8fe5266_1 libpng conda-forge/linux-64::libpng-1.6.37-h21135ba_2 libtiff conda-forge/linux-64::libtiff-4.3.0-hf544144_1 libwebp-base conda-forge/linux-64::libwebp-base-1.2.1-h7f98852_0 libxml2 conda-forge/linux-64::libxml2-2.9.12-h72842e0_0 lz4-c conda-forge/linux-64::lz4-c-1.9.3-h9c3ff4c_1 matplotlib-base conda-forge/linux-64::matplotlib-base-3.3.4-py36hd391965_0 metis conda-forge/linux-64::metis-5.1.0-h58526e2_1006 mpfr conda-forge/linux-64::mpfr-4.1.0-h9202a9a_1 networkx conda-forge/noarch::networkx-2.6.3-pyhd8ed1ab_0 numpy conda-forge/linux-64::numpy-1.19.5-py36hfc0c790_2 olefile conda-forge/noarch::olefile-0.46-pyh9f0ad1d_1 openjpeg conda-forge/linux-64::openjpeg-2.4.0-hb52868f_1 pandas conda-forge/linux-64::pandas-1.1.5-py36h284efc9_0 pillow conda-forge/linux-64::pillow-8.3.2-py36h676a545_0 pyparsing conda-forge/noarch::pyparsing-2.4.7-pyh9f0ad1d_0 python-dateutil conda-forge/noarch::python-dateutil-2.8.2-pyhd8ed1ab_0 python-igraph conda-forge/linux-64::python-igraph-0.9.6-py36h644ed5e_0 python-louvain conda-forge/noarch::python-louvain-0.15-pyhd3deb0d_0 python_abi conda-forge/linux-64::python_abi-3.6-2_cp36m pytz conda-forge/noarch::pytz-2021.3-pyhd8ed1ab_0 scikit-learn conda-forge/linux-64::scikit-learn-0.24.2-py36hc89565f_1 scipy conda-forge/linux-64::scipy-1.5.3-py36h81d768a_1 six conda-forge/noarch::six-1.16.0-pyh6c4a22f_0 suitesparse conda-forge/linux-64::suitesparse-5.10.1-h9e50725_1 tbb conda-forge/linux-64::tbb-2021.3.0-h4bd325d_0 texttable conda-forge/noarch::texttable-1.6.4-pyhd8ed1ab_0 threadpoolctl conda-forge/noarch::threadpoolctl-3.0.0-pyh8a188c0_0 tornado conda-forge/linux-64::tornado-6.1-py36h8f6f2f9_1 zstd conda-forge/linux-64::zstd-1.5.0-ha95c52a_0

Consider to install these (optional) packages to run all possible Giotto commands for spatial analyses: scran MAST smfishHmrf trendsceek SPARK multinet FactoMiner Giotto does not automatically install all these packages as they are not absolutely required and this reduces the number of dependencies no external python path was provided, but a giotto python environment was found and will be used Warning messages: 1: In createGiottoObject(raw_exprs = expr_mat, spatial_locs = bead_positions[, : module: pandas was not found with python path: /home/unix/orr/.local/share/r-miniconda/envs/giotto_env/bin/python

2: In createGiottoObject(raw_exprs = expr_mat, spatial_locs = bead_positions[, : module: igraph was not found with python path: /home/unix/orr/.local/share/r-miniconda/envs/giotto_env/bin/python

3: In createGiottoObject(raw_exprs = expr_mat, spatial_locs = bead_positions[, : module: leidenalg was not found with python path: /home/unix/orr/.local/share/r-miniconda/envs/giotto_env/bin/python

4: In createGiottoObject(raw_exprs = expr_mat, spatial_locs = bead_positions[, : module: sklearn was not found with python path: /home/unix/orr/.local/share/r-miniconda/envs/giotto_env/bin/python


R session.

sessionInfo() R version 4.0.0 (2020-04-24) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Red Hat Enterprise Linux

Matrix products: default BLAS: /usr/lib64/libblas.so.3.4.2 LAPACK: /broad/software/free/Linux/redhat_7_x86_64/pkgs/r_4.0.0/lib64/R/lib/libRlapack.so

locale: [1] C

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] Matrix_1.3-2 data.table_1.14.2 Giotto_1.0.4

loaded via a namespace (and not attached): [1] Rcpp_1.0.7 pillar_1.6.3 compiler_4.0.0 tools_4.0.0
[5] jsonlite_1.7.2 lifecycle_1.0.1 tibble_3.1.5 gtable_0.3.0
[9] lattice_0.20-41 pkgconfig_2.0.3 png_0.1-7 rlang_0.4.11
[13] DBI_1.1.1 parallel_4.0.0 dplyr_1.0.7 generics_0.1.0
[17] vctrs_0.3.8 rappdirs_0.3.3 rprojroot_2.0.2 grid_4.0.0
[21] tidyselect_1.1.1 reticulate_1.22 glue_1.4.2 here_1.0.1
[25] R6_2.5.1 fansi_0.5.0 ggplot2_3.3.5 purrr_0.3.4
[29] blob_1.2.1 magrittr_2.0.1 scales_1.1.1 ellipsis_0.3.2
[33] assertthat_0.2.1 colorspace_2.0-2 utf8_1.2.2 munsell_0.5.0
[37] crayon_1.4.1 


2. Using the conda environment I created.

library(Giotto) library(data.table) library(Matrix) python_path <- "/ahg/regevdata/users/orr/code/miniconda3/envs/giotto_env/bin/python" bead_positions <- fread(file="/broad/regevtmp/orr/slideseq/BeadLocationsForR.csv") expr_matrix<-fread(file="/broad/regevtmp/orr/slideseq/MappedDGEForR.csv") expr_mat <- as.matrix(expr_matrix[,-1]);rownames(expr_mat) = expr_matrix$Row my_instructions <- createGiottoInstructions(python_path = python_path) Slide_test <- createGiottoObject(raw_exprs = expr_mat, spatial_locs = bead_positions[,.(xcoord, ycoord)], instructions = my_instructions) showGiottoInstructions(Slide_test)

Consider to install these (optional) packages to run all possible Giotto commands for spatial analyses: scran MAST smfishHmrf trendsceek SPARK multinet FactoMiner Giotto does not automatically install all these packages as they are not absolutely required and this reduces the number of dependenciesWarning messages: 1: In createGiottoObject(raw_exprs = expr_mat, spatial_locs = bead_positions[, : module: pandas was not found with python path: /ahg/regevdata/users/orr/code/miniconda3/envs/giotto_env/bin/python

2: In createGiottoObject(raw_exprs = expr_mat, spatial_locs = bead_positions[, : module: igraph was not found with python path: /ahg/regevdata/users/orr/code/miniconda3/envs/giotto_env/bin/python

3: In createGiottoObject(raw_exprs = expr_mat, spatial_locs = bead_positions[, : module: leidenalg was not found with python path: /ahg/regevdata/users/orr/code/miniconda3/envs/giotto_env/bin/python

$python_path [1] "/ahg/regevdata/users/orr/code/miniconda3/envs/giotto_env/bin/python"


These three missing packages are all the in the giotto_env conda environment I had created.

(giotto_env) <1019|0>orr@login03 ~$ conda list

packages in environment at /ahg/regevdata/users/orr/code/miniconda3/envs/giotto_env:

#

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
arpack 3.7.0 hdefa2d7_2 conda-forge blas 1.0 mkl
bottleneck 1.3.2 py39hdd57654_1
ca-certificates 2021.5.30 ha878542_0 conda-forge certifi 2021.5.30 py39hf3d152e_0 conda-forge decorator 4.4.2 py_0 anaconda glpk 4.65 h9202a9a_1004 conda-forge gmp 6.2.1 h58526e2_0 conda-forge icu 68.1 h58526e2_0 conda-forge igraph 0.9.4 ha184e22_0 conda-forge intel-openmp 2021.3.0 h06a4308_3350
joblib 1.0.1 pypi_0 pypi ld_impl_linux-64 2.35.1 h7274673_9
leidenalg 0.8.7 py39he80948d_0 conda-forge libblas 3.9.0 11_linux64_mkl conda-forge libcblas 3.9.0 11_linux64_mkl conda-forge libffi 3.3 he6710b0_2
libgcc-ng 9.3.0 h5101ec6_17
libgfortran-ng 11.2.0 h69a702a_9 conda-forge libgfortran5 11.2.0 h5c6108e_9 conda-forge libgomp 9.3.0 h5101ec6_17
libiconv 1.16 h516909a_0 conda-forge liblapack 3.9.0 11_linux64_mkl conda-forge libstdcxx-ng 9.3.0 hd4cf53a_17
libxml2 2.9.12 h72842e0_0 conda-forge metis 5.1.0 h58526e2_1006 conda-forge mkl 2021.3.0 h06a4308_520
mkl-service 2.4.0 py39h7f8727e_0
mkl_fft 1.3.0 py39h42c9631_2
mkl_random 1.2.2 py39h51133e4_0
mpfr 4.1.0 h9202a9a_1 conda-forge ncurses 6.2 he6710b0_1
networkx 2.5 py_0 anaconda numexpr 2.7.3 py39h22e1b3c_1
numpy 1.20.3 py39hf144106_0
numpy-base 1.20.3 py39h74d4b33_0
openssl 1.1.1l h7f8727e_0
pandas 1.3.3 py39h8c16a72_0
pip 21.2.4 py37h06a4308_0
python 3.9.7 h12debd9_1
python-dateutil 2.8.2 pyhd3eb1b0_0
python-igraph 0.9.6 py39hfef886c_0 conda-forge python-louvain 0.15 pyhd3deb0d_0 conda-forge python_abi 3.9 2_cp39 conda-forge pytz 2021.1 pyhd3eb1b0_0
readline 8.1 h27cfd23_0
scikit-learn 1.0 pypi_0 pypi scipy 1.7.1 pypi_0 pypi setuptools 58.0.4 py39h06a4308_0
six 1.16.0 pyhd3eb1b0_0
sqlite 3.36.0 hc218d9a_0
suitesparse 5.10.1 hd8046ac_0 conda-forge tbb 2020.2 h4bd325d_4 conda-forge texttable 1.6.4 pyhd8ed1ab_0 conda-forge threadpoolctl 3.0.0 pypi_0 pypi tk 8.6.11 h1ccaba5_0
tzdata 2021a h5d7bf9c_0
wheel 0.37.0 pyhd3eb1b0_1
xz 5.2.5 h7b6447c_0
zlib 1.2.11 h7b6447c_3

ndelrossi7 commented 3 years ago

Hi @orrzor, thanks for reaching out. Are you able to create the conda environment with anaconda3 as opposed to miniconda3? I am still looking at possible reasons for this but that is 1 thing I see that is different from our tutorial. Thanks!

RubD commented 3 years ago

Hi @orrzor I'm amazed to see that even after a manual installation where all the modules have been installed, you still can not detect them.

One thing to try would be to reinstall the Giotto environment in a fresh R session without loading any other libraries or running any other code. You can simply do:

library(Giotto)  
installGiottoEnvironment(force_miniconda = TRUE, force_environment = TRUE)

This will force the reinstallation of miniconda and re-create the Giotto environment. It could be that the first time you created the Giotto environment the reticulate package was already used by another package and that could lead to problems.

Unfortunately I don't have a (working) linux machine at hand right now, but it does seems to work properly on our linux servers. Let us know if this worked.

orrzor commented 3 years ago

Hi Ruben, thanks for your help! I tried uninstalling and reinstalling Giotto, reticulate, and the giotto_env and was still getting the same error. After a lot of fiddling with reticulate, I found the issue with these specific python packages that weren't loading was due to the LD_LIBRARY_PATH environment variable. When importing modules with reticulate directly, I was seeing this error. 

> import("leidenalg")
Error in py_module_import(module, convert = convert) : 
  ImportError: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /home/unix/orr/.local/share/r-miniconda/envs/giotto_env/lib/python3.6/site-packages/igraph/_igraph.cpython-36m-x86_64-linux-gnu.so)

and found I could fix this by running the following before starting my R from the terminal.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/unix/orr/.local/share/r-miniconda/envs/giotto_env/lib

Now things work for me in the environment. So again, I think this is another reticulate issue, and may be worth highlighting on your installation instructions. Thanks again fro your help! Best- Orr

RubD commented 2 years ago

Thanks @orrzor for letting us know how to fix this issue. I tried looking for an option that could work directly within R/Rstudio, but that doesn't seem to be trivial. We'll make sure to update our installation section with your suggestion in the next couple of days. Thanks, Ruben

orrzor commented 2 years ago

thank you Ruben! I'm excited to see what I get with the HMRF now that the technical issues (fingers crossed) are resolved. best, Orr