search

drieslab / Giotto

Spatial omics analysis toolbox
https://drieslab.github.io/Giotto_website/
Other
258 stars 98 forks source link

another issue when doing doLeidenCluster(): #425

Closed RubD closed 1 year ago

RubD commented 1 year ago 
    I also got another issue when doing doLeidenCluster():

Error in py_run_file_impl(file, local, convert) : ImportError: DLL load failed while importing _c_leiden: The specified procedure could not be found.

Originally posted by @GCatatGC in https://github.com/drieslab/Giotto/issues/421#issuecomment-1319233849

GCatatGC commented 1 year ago

Thanks for your help. For more details of my issue, I installed Giotto on a new PC (win11) this week. R version: 4.2.2 and I ran the installation code from the latest version https://giottosuite.readthedocs.io/en/latest/gettingstarted.html

devtools::install_github("drieslab/Giotto@suite") library(Giotto) installGiottoEnvironment() and got an error which I solved as the others mentioned: Error: giotto environment was expected, but NOT found at C:/Users/bwhcc/AppData/Local/r-miniconda\envs\giotto_env\python.exe |---- install giotto environment ----|

CondaSSLError: OpenSSL appears to be unavailable on this machine. OpenSSL is required to download and install packages.

Exception: HTTPSConnectionPool(host='conda.anaconda.org', port=443): Max retries exceeded with url: /conda-forge/win-64/current_repodata.json (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available."))

Error: Error creating conda environment 'giotto_env' [exit code 1]

I solved by: Solve the problem: 1.Go to minicoda file path and Copy existing dlls from \bin to \DLLs
Copy libcrypto-1_1-x64.dll and libssl-1_1-x64.dll from C:\Users\bwhcc\AppData\Local\r-miniconda\Library\bin to C:\Users\bwhcc\AppData\Local\r-miniconda\DLLs

Then I ran the code in CosMx Tutorial: https://giottosuite.readthedocs.io/en/latest/subsections/datasets/Nanostring_Lung12_jan26_21.html

And then get the error from doLeidenCluster in chapter 10. Cluster.

mattobny commented 1 year ago

Yes, that is a proper workaround for the OpenSSL error. It should be noted that this is a conda/miniconda installation problem, not a problem with Giotto. Conda developers are aware of this, but unfortunately there does not seem to be any headway on fixing this issue, workaround outstanding.

I also just did a fresh install on a brand new win11 machine, and have yet to experience any issue. To assist you, I just need some more details. Can you provide the following:

@GCatatGC

GCatatGC commented 1 year ago

Yes, that is a proper workaround for the OpenSSL error. It should be noted that this is a conda/miniconda installation problem, not a problem with Giotto. Conda developers are aware of this, but unfortunately there does not seem to be any headway on fixing this issue, workaround outstanding.

I also just did a fresh install on a brand new win11 machine, and have yet to experience any issue. To assist you, I just need some more details. Can you provide the following:

  • The entire error message from doLeidenCluster()
  • The outputs of the commands reticulate::py_config() and reticulate::py_list_packages() after running reticulate::use_python("C:/Users/bwhcc/AppData/Local/r-miniconda\envs\giotto_env\python.exe") (or your alternative, desired python path for use with Giotto)

@GCatatGC

Hi mattobny, Thanks for your reply. The entire error message is " Error in py_run_file_impl(file, local, convert) : ImportError: DLL load failed while importing _c_leiden: The specified procedure could not be found."

The outputs are:

reticulate::py_config() python: C:/Users/bwhcc/AppData/Local/r-miniconda/envs/giotto_env/python.exe libpython: C:/Users/bwhcc/AppData/Local/r-miniconda/envs/giotto_env/python310.dll pythonhome: C:/Users/bwhcc/AppData/Local/r-miniconda/envs/giotto_env version: 3.10.2 | packaged by conda-forge | (main, Mar 8 2022, 15:47:33) [MSC v.1929 64 bit (AMD64)] Architecture: 64bit numpy: C:/Users/bwhcc/AppData/Local/r-miniconda/envs/giotto_env/Lib/site-packages/numpy numpy_version: 1.23.4

NOTE: Python version was forced by use_python function

reticulate::py_list_packages() package version requirement channel 1 bzip2 1.0.8 bzip2=1.0.8 conda-forge 2 ca-certificates 2022.10.11 ca-certificates=2022.10.11 pkgs/main 3 contourpy 1.0.6 contourpy=1.0.6 pypi 4 cycler 0.11.0 cycler=0.11.0 pypi 5 fonttools 4.38.0 fonttools=4.38.0 pypi 6 glpk 5.0 glpk=5.0 conda-forge 7 igraph 0.10.2 igraph=0.10.2 conda-forge 8 intel-openmp 2022.1.0 intel-openmp=2022.1.0 conda-forge 9 joblib 1.2.0 joblib=1.2.0 conda-forge 10 kiwisolver 1.4.4 kiwisolver=1.4.4 pypi 11 leidenalg 0.9.0 leidenalg=0.9.0 conda-forge 12 libblas 3.9.0 libblas=3.9.0 conda-forge 13 libcblas 3.9.0 libcblas=3.9.0 conda-forge 14 libffi 3.4.2 libffi=3.4.2 conda-forge 15 libiconv 1.17 libiconv=1.17 conda-forge 16 liblapack 3.9.0 liblapack=3.9.0 conda-forge 17 liblapacke 3.9.0 liblapacke=3.9.0 conda-forge 18 libsqlite 3.40.0 libsqlite=3.40.0 conda-forge 19 libxml2 2.10.3 libxml2=2.10.3 conda-forge 20 libzlib 1.2.13 libzlib=1.2.13 conda-forge 21 m2w64-gcc-libgfortran 5.3.0 m2w64-gcc-libgfortran=5.3.0 conda-forge 22 m2w64-gcc-libs 5.3.0 m2w64-gcc-libs=5.3.0 conda-forge 23 m2w64-gcc-libs-core 5.3.0 m2w64-gcc-libs-core=5.3.0 conda-forge 24 m2w64-gmp 6.1.0 m2w64-gmp=6.1.0 conda-forge 25 m2w64-libwinpthread-git 5.0.0.4634.697f757 m2w64-libwinpthread-git=5.0.0.4634.697f757 conda-forge 26 matplotlib 3.6.2 matplotlib=3.6.2 pypi 27 mkl 2022.1.0 mkl=2022.1.0 conda-forge 28 mpir 3.0.0 mpir=3.0.0 conda-forge 29 msys2-conda-epoch 20160418 msys2-conda-epoch=20160418 conda-forge 30 networkx 2.8.8 networkx=2.8.8 conda-forge 31 numpy 1.23.4 numpy=1.23.4 conda-forge 32 openssl 3.0.7 openssl=3.0.7 conda-forge 33 packaging 21.3 packaging=21.3 pypi 34 pandas 1.5.1 pandas=1.5.1 conda-forge 35 pillow 9.3.0 pillow=9.3.0 pypi 36 pip 22.3.1 pip=22.3.1 conda-forge 37 pyparsing 3.0.9 pyparsing=3.0.9 pypi 38 python 3.10.2 python=3.10.2 conda-forge 39 python-dateutil 2.8.2 python-dateutil=2.8.2 conda-forge 40 python-igraph 0.10.2 python-igraph=0.10.2 conda-forge 41 python-louvain 0.16 python-louvain=0.16 pypi 42 python_abi 3.10 python_abi=3.10 conda-forge 43 pytz 2022.6 pytz=2022.6 conda-forge 44 scikit-learn 1.1.3 scikit-learn=1.1.3 conda-forge 45 scipy 1.9.3 scipy=1.9.3 conda-forge 46 seaborn 0.12.1 seaborn=0.12.1 pypi 47 setuptools 65.5.1 setuptools=65.5.1 conda-forge 48 six 1.16.0 six=1.16.0 conda-forge 49 smfishhmrf 1.3.3 smfishhmrf=1.3.3 pypi 50 sqlite 3.40.0 sqlite=3.40.0 conda-forge 51 suitesparse 5.4.0 suitesparse=5.4.0 conda-forge 52 tbb 2021.7.0 tbb=2021.7.0 conda-forge 53 texttable 1.6.4 texttable=1.6.4 conda-forge 54 threadpoolctl 3.1.0 threadpoolctl=3.1.0 conda-forge 55 tk 8.6.12 tk=8.6.12 conda-forge 56 tzdata 2022f tzdata=2022f conda-forge 57 ucrt 10.0.22621.0 ucrt=10.0.22621.0 conda-forge 58 vc 14.3 vc=14.3 conda-forge 59 vs2015_runtime 14.32.31332 vs2015_runtime=14.32.31332 conda-forge 60 wheel 0.38.4 wheel=0.38.4 conda-forge 61 xz 5.2.6 xz=5.2.6 conda-forge

Is there any possibility that leidenalg depends on C++ or what but I did not install properly?

GCatatGC commented 1 year ago

Yes, that is a proper workaround for the OpenSSL error. It should be noted that this is a conda/miniconda installation problem, not a problem with Giotto. Conda developers are aware of this, but unfortunately there does not seem to be any headway on fixing this issue, workaround outstanding.

I also just did a fresh install on a brand new win11 machine, and have yet to experience any issue. To assist you, I just need some more details. Can you provide the following:

  • The entire error message from doLeidenCluster()
  • The outputs of the commands reticulate::py_config() and reticulate::py_list_packages() after running reticulate::use_python("C:/Users/bwhcc/AppData/Local/r-miniconda\envs\giotto_env\python.exe") (or your alternative, desired python path for use with Giotto)

@GCatatGC

OK. After install Visual C++ I solved this issue. Thanks for all of you!

RubD commented 1 year ago

Seems like this is resolved. We can open it again if anything else comes up.

GCatatGC commented 1 year ago

Yes, that is a proper workaround for the OpenSSL error. It should be noted that this is a conda/miniconda installation problem, not a problem with Giotto. Conda developers are aware of this, but unfortunately there does not seem to be any headway on fixing this issue, workaround outstanding. I also just did a fresh install on a brand new win11 machine, and have yet to experience any issue. To assist you, I just need some more details. Can you provide the following:

  • The entire error message from doLeidenCluster()
  • The outputs of the commands reticulate::py_config() and reticulate::py_list_packages() after running reticulate::use_python("C:/Users/bwhcc/AppData/Local/r-miniconda\envs\giotto_env\python.exe") (or your alternative, desired python path for use with Giotto)

@GCatatGC

OK. After install Visual C++ I solved this issue. Thanks for all of you! For somebody also a rookie just like me, wanted to install Giotto on a new PC and ran into similar issue: Actually, I solved this issue by using manual python3 environment. instead of miniconda3. I guess miniconda3 group made some mistakes. Here is my workflow to install Giotto on a new PC:

`

Firstly, install R (4.2.0 or 4.2.2), Rstudio, Rtools, Visual C++;

Secondly, install Giotto with the code in Rstudio:

install.packages('devtools') library(devtools) devtools::install_github("drieslab/Giotto@suite")

Third, install python-3.8.7-amd64.exe (choose “Add PATH” when asked, use the default installation path):

Then, press win+R buttons on your keyboard, type in "cmd" and Enter

type in" pip3 install pandas python-igraph networkx python-louvain leidenalg scikit-learn smfishHmrf " and Enter

Go to the path and find the python.exe: such like "c:/users/your_own_user_name/appdata/local/programs/python/python38/python.exe"

When ever you want to use Giotto, type the following code to set the python_path :

library(Giotto) my_instructions = createGiottoInstructions(python_path = 'c:/users/your_own_user_name/appdata/local/programs/python/python38/python.exe') my_giotto_object = createGiottoObject( raw_exprs = '...', spatial_locs = '...', instructions = my_instructions) `

kguion1 commented 1 year ago

Hi @mattobny! I am having a similar issue with leidenalg. The doLeidenCluster function returns this error:

Error in py_run_file_impl(path.expand(file), local, convert) : ImportError: DLL load failed: The specified procedure could not be found.

I tried the suggested solutions above, and all packages except leidenalg are being found by my system. I tried reticulate::py_install('leidenalg') and it looked like it installed; however when I run the py_module_available('leidenalg') it is still false. I realize this may be an issue for reticulate, but I was hoping you may have some suggestions for Giotto specifically.

image

I also opened my giotto python.exe and there are no error messages or warnings when I open it in terminal.

Also, is the python_path supposed to include the python.exe file name or just the folder it is in?

Any help is greatly appreciated!

mattobny commented 1 year ago

Hello @kguion1 what is your Windows OS version? What version of Giotto are you running?

In the meantime, I will provide some guidance assuming you are up-to-date (Giotto v3.3). First, the python_path argument should be the full path to your python executable, i.e. C:/Users/etc/giotto_env/python.exe. If you intend to use the conda environment installed by installGiottoEnviornment() or that you may have installed manually with the .yml configuration files, then Giotto will activate this environment by default. If you are unsure where this lives, you can call checkGiottoEnviornment() which will print out the path to the python executable in the Giotto environment if it is found.

Try opening a fresh R session and setting reticulate to your python environment directly by calling reticulate::use_python("your/path/to/python.exe") before loading Giotto. Please follow up if you have any other questions.

kguion1 commented 1 year ago

Okay thank you! I actually figured out that it works if I run RStudio as an administrator. Here the redticulate::use_python() suggestion works; however, in regular RStudio it still doesn't. Thanks for a great package!

mattobny commented 1 year ago

Glad to hear, thank you for following up!

kguion1 commented 1 year ago

For anyone who needs more assistance on leidenalg and reticulate::py_module_available('leidenalg') giving False this is a good stack overflow: https://stackoverflow.com/questions/50145643/unable-to-change-python-path-in-reticulate/58354084#58354084

I found that the Tools --> Global Options --> Python --> Select interpretor --> Conda environment --> find the giotto one to be effective!

Zockimonster commented 4 months ago

In one of my very early versions of Giotto the leiden clustering worked fine, but the image and the spots in my visium data set didn't match, so I decided to update the version (now 4.0.8). After that the image and the spots matched, but now the leiden clustering doesn't work - just the way as described above. I already tried with an earlier version of conda and of reticulate, but either nothing works, or I get this Error when doing the leiden clustering: 'Error in py_run_file_impl(file, local, convert) : ImportError: DLL load failed while importing _c_leiden: The specified procedure could not be found.'

If I try to get information about the error massage, I get this: 'reticulate::py_last_error()

── Python Exception Message ──────────────────────────────────────────────────── Traceback (most recent call last): File "C:/Users/X/AppData/Local/R/win-library/4.3/Giotto/python/python_leiden.py", line 10, in import leidenalg as la File "C:\Users\X\AppData\Local\R\win-library\4.3\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook return _run_hook(name, _hook) File "C:\Users\X\AppData\Local\R\win-library\4.3\reticulate\python\rpytools\loader.py", line 93, in _run_hook module = hook() File "C:\Users\X\AppData\Local\R\win-library\4.3\reticulate\python\rpytools\loader.py", line 117, in _hook return _find_andload(name, import) File "C:\Users\X\AppData\Local\R-MINI~1\envs\GIOTTO~1\lib\site-packages\leidenalg__init__.py", line 35, in from .functions import ALL_COMMS File "C:\Users\X\AppData\Local\R\win-library\4.3\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook return _run_hook(name, _hook) File "C:\Users\X\AppData\Local\R\win-library\4.3\reticulate\python\rpytools\loader.py", line 93, in _run_hook module = hook() File "C:\Users\X\AppData\Local\R\win-library\4.3\reticulate\python\rpytools\loader.py", line 117, in _hook return _find_andload(name, import) File "C:\Users\X\AppData\Local\R-MINI~1\envs\GIOTTO~1\lib\site-packages\leidenalg\functions.py", line 3, in from ._c_leiden import ALL_COMMS File "C:\Users\X\AppData\Local\R\win-library\4.3\reticulate\python\rpytools\loader.py", line 119, in _find_and_load_hook return _run_hook(name, _hook) File "C:\Users\X\AppData\Local\R\win-library\4.3\reticulate\python\rpytools\loader.py", line 93, in _run_hook module = hook() File "C:\Users\X\AppData\Local\R\win-library\4.3\reticulate\python\rpytools\loader.py", line 117, in _hook return _find_andload(name, import) ImportError: DLL load failed while importing _c_leiden: The specified procedure could not be found.

── R Traceback ───────────────────────────────────────────────────────────────── ▆

  1. └─Giotto::doLeidenCluster(...)
  2. └─reticulate::source_python(file = python_leiden_function)
  3. └─reticulate::py_run_file(file, local = FALSE, convert = convert)
  4. └─reticulate:::py_run_file_impl(file, local, convert)'

I already changed the Python Interpreter in Global Options, ran RStudio (current version 2024.04.1+748) as administrator, tried it in R (version 4.3.0), tried it using previous versions of leidenalg, but I still can't run the doLeidenCluster() function since I always get the same error... And I'm just no informatician... I liked to Leiden cluster plots, so I would appreciate any idea, how I could get the Leiden clustering work ; ) Have a nice weekend! Best regards Zockimonster

erik40021 commented 4 months ago

I'm having the exact same error when trying to run doLeidenCluster(). Before, for every Giotto function I use, I also get the following warning:

In .check_giotto_python_modules(my_python_path = instructions(.Object,  :
  module: leidenalg was not found with python path: C:/Users/schuefte/AppData/Local/r-miniconda\envs\giotto_env\python.exe

Tried out everything that is suggested here, with no success. Giotto version is 4.0.5. Please help :/

josschavezf commented 4 months ago

Have you tried running: reticulate::conda_install(envname = 'giotto_env', packages = 'leidenalg', pip = TRUE)

RubD commented 4 months ago

I also want to add here that we also have a leiden clustering function that doesn't require Python. It works the same way and typically gives very similar results. It's called doLeidenClusterIgraph().

erik40021 commented 4 months ago

Thanks a lot for the comments, I did try to (re-)install it manually including your command @josschavezf. The problem is, the leidenalg package is already there from the beginning. The installation seems to work perfectly fine for every package, I see it both in the Windows folder as well when checking via _reticulate::py_listpackages() in R. Going for the alternative Leiden-Clustering function is fine when using the newest Giotto version I guess...