search

TheoreticalEcology / s-jSDM

Scalable joint species distribution modeling
https://cran.r-project.org/web/packages/sjSDM/index.html
GNU General Public License v3.0
68 stars 14 forks source link

installation error in macOS #78

Closed dougwyu closed 2 years ago

dougwyu commented 3 years ago

Hi, I'm reinstalling R and everything following a migration to an M1 Mac.

The following code did not work for installation, and it looks like the sjSDM_env is incomplete?

doug

`

in terminal

conda create --name sjSDM_env python=3.7 # choose py3 conda activate sjSDM_env # now in py3 env conda install pytorch torchvision cpuonly -c pytorch # cpu

in R

devtools::install_github("https://github.com/TheoreticalEcology/s-jSDM", subdir = "sjSDM") library(sjSDM) packageVersion("sjSDM") # [1] ‘0.1.8’ set.seed(42) community <- simulate_SDM(sites = 100, species = 10, env = 3, se = TRUE) Env <- community$env_weights Occ <- community$response SP <- matrix(rnorm(200, 0, 0.3), 100, 2) # spatial coordinates (no effect on species occurences)

model <- sjSDM(Y = Occ, env = linear(data = Env, formula = ~X1+X2+X3), spatial = linear(data = SP, formula = ~0+X1:X2), se = TRUE, family=binomial("probit"), sampling = 100L) `

── Attaching sjSDM ────────────────────────────────────────────── 0.1.8 ── x torch x torch_optimizer x pyro x madgrad

Torch or other dependencies not found:

  1. Use install_sjSDM() to install Pytorch and conda automatically
  2. Installation trouble shooting guide: ?installation_help
  3. If 1) and 2) did not help, please create an issue on https://github.com/TheoreticalEcology/s-jSDM/issues (see ?install_diagnostic) Error: PyTorch not installed

install_diagnostic()

EnvironmentLocationNotFound: Not a conda environment: /Users/Negorashi2011/opt/miniconda3/envs/miniconda3

EnvironmentLocationNotFound: Not a conda environment: /Users/Negorashi2011/opt/miniconda3/envs/miniconda3

    name

1 miniconda3 2 sjSDM_env python 1 /Users/Negorashi2011/opt/miniconda3/bin/python 2 /Users/Negorashi2011/opt/miniconda3/envs/sjSDM_env/bin/python

ENV: miniconda3

torch:

numpy:

ENV: sjSDM_env

torch:

packages in environment at /Users/Negorashi2011/opt/miniconda3/envs/sjSDM_env:

#

Name Version Build Channel

pytorch 1.9.1 py3.7_0 pytorch torchvision 0.10.1 py37_cpu pytorch

numpy:

packages in environment at /Users/Negorashi2011/opt/miniconda3/envs/sjSDM_env:

#

Name Version Build Channel

python: /usr/bin/python libpython: /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib pythonhome: /System/Library/Frameworks/Python.framework/Versions/2.7:/System/Library/Frameworks/Python.framework/Versions/2.7 version: 2.7.16 (default, Aug 30 2021, 14:43:11) [GCC Apple LLVM 12.0.5 (clang-1205.0.19.59.6) [+internal-os, ptrauth-isa=deploy numpy: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy numpy_version: 1.8.0 torch: [NOT FOUND]

python versions found: /usr/bin/python3 /usr/bin/python

 active environment : None
   user config file : /Users/Negorashi2011/.condarc

populated config files : /Users/Negorashi2011/.condarc conda version : 4.10.3 conda-build version : not installed python version : 3.9.5.final.0 virtual packages : osx=10.16=0 unix=0=0 __archspec=1=x86_64 base environment : /Users/Negorashi2011/opt/miniconda3 (writable) conda av data dir : /Users/Negorashi2011/opt/miniconda3/etc/conda conda av metadata url : None channel URLs : https://repo.anaconda.com/pkgs/main/osx-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/osx-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /Users/Negorashi2011/opt/miniconda3/pkgs /Users/Negorashi2011/.conda/pkgs envs directories : /Users/Negorashi2011/opt/miniconda3/envs /Users/Negorashi2011/.conda/envs platform : osx-64 user-agent : conda/4.10.3 requests/2.25.1 CPython/3.9.5 Darwin/20.6.0 OSX/10.16 UID:GID : 502:20 netrc file : None offline mode : False

MaximilianPi commented 3 years ago

Hi Doug, yes, you're right, the environment is not complete and might be even missing. Strange. Try this:

  1. rerun the sjSDM_install() function
  2. restart RStudio
  3. if the error persists: a) reticulate::conda_create('r-reticulate', python_version='3.6') b) rerun sjSDM_install() c) reticulate::conda_install('r-reticulate',packages = c("pytorch", "torchvision", "torchaudio" ,"-c", "pytorch")) d) reticulate::conda_install('r-reticulate',packages = c("pyro-ppl", "torch_optimizer", "madgrad"), pip=TRUE)
dougwyu commented 3 years ago

Thanks for this. I ran commands a-d, and i still get this error when i run this:

`

model <- sjSDM(Y = Occ, env = linear(data = Env, formula = ~X1+X2+X3), spatial = linear(data = SP, formula = ~0+X1:X2), se = TRUE, family=binomial("probit"), sampling = 100L) ── Attaching sjSDM ────────────────────────────────────────────── 0.1.8 ── x torch x torch_optimizer x pyro x madgrad

Torch or other dependencies not found:

  1. Use install_sjSDM() to install Pytorch and conda automatically
  2. Installation trouble shooting guide: ?installation_help
  3. If 1) and 2) did not help, please create an issue on https://github.com/TheoreticalEcology/s-jSDM/issues (see ?install_diagnostic) Error: PyTorch not installed `

I think i have a clue about the problem. when i run python -V in my terminal, i get Python 3.9.5, but when i run it in the Terminal inside RStudio, i get Python 2.7.16, which is the default Apple Python installation.

This is very strange!

`

sjSDM::install_diagnostic()

EnvironmentLocationNotFound: Not a conda environment: /Users/Negorashi2011/opt/miniconda3/envs/miniconda3

EnvironmentLocationNotFound: Not a conda environment: /Users/Negorashi2011/opt/miniconda3/envs/miniconda3

      name

1 miniconda3 2 r-reticulate python 1 /Users/Negorashi2011/opt/miniconda3/bin/python 2 /Users/Negorashi2011/opt/miniconda3/envs/r-reticulate/bin/python

ENV: miniconda3

torch:

numpy:

ENV: r-reticulate

torch:

packages in environment at /Users/Negorashi2011/opt/miniconda3/envs/r-reticulate:

#

Name Version Build Channel

pytorch 1.9.1 py3.6_0 pytorch pytorch-ranger 0.1.1 pypi_0 pypi torch-optimizer 0.1.0 pypi_0 pypi torchaudio 0.9.1 py36 pytorch torchvision 0.10.1 py36_cpu pytorch

numpy:

packages in environment at /Users/Negorashi2011/opt/miniconda3/envs/r-reticulate:

#

Name Version Build Channel

numpy 1.17.0 py36hacdab7b_0
numpy-base 1.17.0 py36h6575580_0

python: /usr/bin/python libpython: /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib pythonhome: /System/Library/Frameworks/Python.framework/Versions/2.7:/System/Library/Frameworks/Python.framework/Versions/2.7 version: 2.7.16 (default, Aug 30 2021, 14:43:11) [GCC Apple LLVM 12.0.5 (clang-1205.0.19.59.6) [+internal-os, ptrauth-isa=deploy numpy: /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy numpy_version: 1.8.0 torch: [NOT FOUND]

python versions found: /usr/bin/python3 /usr/bin/python

 active environment : None
   user config file : /Users/Negorashi2011/.condarc

populated config files : /Users/Negorashi2011/.condarc conda version : 4.10.3 conda-build version : not installed python version : 3.9.5.final.0 virtual packages : osx=10.16=0 unix=0=0 __archspec=1=x86_64 base environment : /Users/Negorashi2011/opt/miniconda3 (writable) conda av data dir : /Users/Negorashi2011/opt/miniconda3/etc/conda conda av metadata url : None channel URLs : https://repo.anaconda.com/pkgs/main/osx-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/osx-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /Users/Negorashi2011/opt/miniconda3/pkgs /Users/Negorashi2011/.conda/pkgs envs directories : /Users/Negorashi2011/opt/miniconda3/envs /Users/Negorashi2011/.conda/envs platform : osx-64 user-agent : conda/4.10.3 requests/2.25.1 CPython/3.9.5 Darwin/20.6.0 OSX/10.16 UID:GID : 502:20 netrc file : None offline mode : False `

MaximilianPi commented 3 years ago

okay, let's try this:

  1. remove miniconda, e.g. by (in terminal) $rm -rf /Users/Negorashi2011/opt/miniconda3
  2. restart Rstudio
  3. sjSDM setup (within R): a)reticulate::install_miniconda() b) reticulate::conda_create('r-reticulate', python_version='3.7') c) rerun sjSDM_install() d) reticulate::conda_install('r-reticulate',packages = c("pytorch", "torchvision", "torchaudio" ,"-c", "pytorch")) e) reticulate::conda_install('r-reticulate',packages = c("pyro-ppl", "torch_optimizer", "madgrad"), pip=TRUE)
  4. reticulate::use_condaenv("r-reticulate")
dougwyu commented 3 years ago

sadly did not work.

let's revisit this in Leipzig?

doug

On 4 Oct 2021, at 12:55, MaximilianPi @.***> wrote:

okay, let's try this:

remove miniconda, e.g. by (in terminal) $rm -rf /Users/Negorashi2011/opt/miniconda3 restart Rstudio sjSDM setup (within R): a) reticulate::conda_create('r-reticulate', python_version='3.7') b) rerun sjSDM_install() c) reticulate::conda_install('r-reticulate',packages = c("pytorch", "torchvision", "torchaudio" ,"-c", "pytorch")) d) reticulate::conda_install('r-reticulate',packages = c("pyro-ppl", "torch_optimizer", "madgrad"), pip=TRUE) reticulate::use_condaenv("r-reticulate") — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/TheoreticalEcology/s-jSDM/issues/78#issuecomment-933409510, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADB4EP3NKIGXDLE72S244L3UFGIZTANCNFSM5FGTDXIQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

MaximilianPi commented 3 years ago

Oh no, we have to find a way to simplify the installation...

Yes, let's revisit it next week. One last question, can you please post the output of sessionInfo()?

dougwyu commented 3 years ago

by the way, i'm using install_sjSDM() instead of sjSDM_install()

sessionInfo() R version 4.1.1 (2021-08-10) Platform: aarch64-apple-darwin20 (64-bit) Running under: macOS Big Sur 11.6

Matrix products: default LAPACK: /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRlapack.dylib

locale: [1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] sjSDM_0.1.8

loaded via a namespace (and not attached): [1] nlme_3.1-153 fs_1.5.0 usethis_2.0.1
[4] ordinal_2019.12-10 devtools_2.4.2 insight_0.14.4
[7] rprojroot_2.0.2 numDeriv_2016.8-1.1 tools_4.1.1
[10] backports_1.2.1 utf8_1.2.2 R6_2.5.1
[13] sjlabelled_1.1.8 DBI_1.1.1 colorspace_2.0-2
[16] withr_2.4.2 gridExtra_2.3 tidyselect_1.1.1
[19] prettyunits_1.1.1 mnormt_2.0.2 processx_3.5.2
[22] compiler_4.1.1 git2r_0.28.0 cli_3.0.1
[25] desc_1.4.0 scales_1.1.1 checkmate_2.0.0
[28] mvtnorm_1.1-2 psych_2.1.9 callr_3.7.0
[31] stringr_1.4.0 digest_0.6.28 minqa_1.2.4
[34] pkgconfig_2.0.3 parallelly_1.28.1 lme4_1.1-27.1
[37] sessioninfo_1.1.1 fastmap_1.1.0 rlang_0.4.11
[40] rstudioapi_0.13 generics_0.1.0 jsonlite_1.7.2
[43] dplyr_1.0.7 zip_2.2.0 magrittr_2.0.1
[46] Matrix_1.3-4 Rcpp_1.0.7 munsell_0.5.0
[49] fansi_0.5.0 reticulate_1.22 lifecycle_1.0.1
[52] furrr_0.2.3 ucminf_1.1-4 stringi_1.7.4
[55] MASS_7.3-54 pkgbuild_1.2.0 plyr_1.8.6
[58] grid_4.1.1 parallel_4.1.1 listenv_0.8.0
[61] sjmisc_2.8.7 crayon_1.4.1 lattice_0.20-45
[64] splines_4.1.1 tmvnsim_1.0-2 ps_1.6.0
[67] pillar_1.6.3 boot_1.3-28 reshape2_1.4.4
[70] codetools_0.2-18 pkgload_1.2.2 glue_1.4.2
[73] data.table_1.14.2 remotes_2.4.1 BiocManager_1.30.16 [76] png_0.1-7 vctrs_0.3.8 nloptr_1.2.2.2
[79] testthat_3.0.4 gtable_0.3.0 purrr_0.3.4
[82] tidyr_1.1.4 future_1.22.1 assertthat_0.2.1
[85] cachem_1.0.6 ggplot2_3.3.5 xfun_0.26
[88] openxlsx_4.2.4 hablar_0.3.0 broom_0.7.9
[91] tibble_3.1.5 tinytex_0.34 memoise_2.0.0
[94] globals_0.14.0 ellipsis_0.3.2 here_1.0.1

sjSDM::install_diagnostic()

EnvironmentLocationNotFound: Not a conda environment: /Users/Negorashi2011/opt/anaconda3/envs/anaconda3

EnvironmentLocationNotFound: Not a conda environment: /Users/Negorashi2011/opt/anaconda3/envs/anaconda3

      name

1 anaconda3 2 r-reticulate python 1 /Users/Negorashi2011/opt/anaconda3/bin/python 2 /Users/Negorashi2011/opt/anaconda3/envs/r-reticulate/bin/python

ENV: anaconda3

torch:

numpy:

ENV: r-reticulate

torch:

packages in environment at /Users/Negorashi2011/opt/anaconda3/envs/r-reticulate:

#

Name Version Build Channel

pytorch 1.9.1 py3.7_0 pytorch pytorch-ranger 0.1.1 pypi_0 pypi torch-optimizer 0.1.0 pypi_0 pypi torchaudio 0.9.1 py37 pytorch torchvision 0.10.1 py37_cpu pytorch

numpy:

packages in environment at /Users/Negorashi2011/opt/anaconda3/envs/r-reticulate:

#

Name Version Build Channel

numpy 1.20.3 py37h4b4dc7a_0
numpy-base 1.20.3 py37he0bd621_0

Error in py_initialize(config$python, config$libpython, config$pythonhome, : /Users/Negorashi2011/opt/anaconda3/envs/r-reticulate/lib/libpython3.7m.dylib - dlopen(/Users/Negorashi2011/opt/anaconda3/envs/r-reticulate/lib/libpython3.7m.dylib, 10): no suitable image found. Did find: /Users/Negorashi2011/opt/anaconda3/envs/r-reticulate/lib/libpython3.7m.dylib: mach-o, but wrong architecture /Users/Negorashi2011/opt/anaconda3/envs/r-reticulate/lib/libpython3.7m.dylib: mach-o, but wrong architecture

MaximilianPi commented 3 years ago

Ah, I found the issue. You have installed the arm64 R-version (Platform: aarch64-apple-darwin20 (64-bit)) which is not compatible with the x86 python stuff. I'm still running the x86 R-version because many R packages and python packages are not yet compatible with arm.

So in short, to run sjSDM (and other deep learning frameworks) you need to install the x86 R-version (https://cran.r-project.org/bin/macosx/) (using the x86 instead of the arm version has (almost) no negative side effects)

florianhartig commented 3 years ago

I haven't tried, but can you run them side by side and switch when needed?

MaximilianPi commented 3 years ago

Hm, switch when needed... Theoretically yes (it's the same situation when you have different R versions installed), but you need always to change first the R path variable so that RStudio starts with the intended R-version (I guess it's different for each OS...)

dougwyu commented 3 years ago

i just installed the Intel version of R, but i got this when trying to install sjSDM

Warning message: In file.symlink(target, symlink) : cannot symlink '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config' to '/var/folders/3y/gxvqrxx968j9b8f6_4vcg7vm0000gp/T/rstudio-fallback-library-path-y31BVP', reason 'File exists'

let's debug this in person?

my computer is brand new, and we can use this computer as a test machine for a simpler installation script.