search

greta-dev / greta

simple and scalable statistical modelling in R
https://greta-stats.org
Other
528 stars 63 forks source link

Use new reticulate python dependencies system #357

Closed jeffreypullin closed 2 years ago

jeffreypullin commented 3 years ago

Hi Nick,

I've been using a little but of {reticulate} for my MSc thesis research and I stumbled across this page in the docs:

https://rstudio.github.io/reticulate/articles/python_dependencies.html

It describes a new system for managing the python dependencies of an R package

Given that installing all the python bits of {greta} is one of the major pain points for users I think it would be worth looking into whether {greta} could benefit from this new system.

Best, Jeffrey

goldingn commented 3 years ago

OMG, this looks like exactly what I've been hoping for! Will put this at the top of the wishlist for the new RSE to try out :)

flyaflya commented 3 years ago

Hi Nick. My MAC users are having installation issues due to this auto-configuration with greta. Might this fix be in one of the dev branches that we can switch over to? My workaround is to downgrade reticulate at the moment.

njtierney commented 3 years ago

Hi @flyaflya - have you tried installing with these instructions: https://forum.greta-stats.org/t/installing-greta-new-script-to-overcome-reticulates-auto-configure/259/2

I've found that worked well for me on my MAC.

goldingn commented 3 years ago

@njtierney: flyaflya on GitHub is the same person (Adam) as ajf on the forum. So I'm guessing Adam has tried his own instructions!

@flyaflya are your Mac users having the same issue even following your instructions from that post?

We don't have a branch with this implemented yet, though Nick T is thinking about this stuff now, so hopefully we can find a fix

njtierney commented 3 years ago

Oh haha! Thanks @goldingn - I'll make sure to check the github users name next time 😅

I'm looking at this now, I'll start testing out the new Config/reticulate: field in the DESCRIPTION file.

flyaflya commented 3 years ago

All these platform handles for me and none of them match :-). LOL! Twice in two days for me that this has been an issue. According to https://namechk.com/ , I would need to pick a completely new handle to span them all.

Yes, I have tried my own instructions and the latest on the greta forum work really well on both MAC and PC as far as I can tell. It took me a little while to realize that reticulate's autoconfigure was the culprit of my issues as I am not a MAC user and the autoconfigure does not seem to be triggerred on a PC.

FYI, I vacillate on what conda environment to install into. For now though, by using the r-reticulate environment for the greta setup, users mostly avoid having to change the .Rprofile to include use_condaenv(...). I've noticed that if the Python environment disappears and the .Rprofile still has the use_condaenv() line, then RStudio will crash on startup without a helpful error message.... so now I am scared to edit .Rprofile's.

njtierney commented 3 years ago

Ah indeed! Names are hard.

That's an interesting error with .Rprofile - I think that Rstudio would be interested in trying to fix that, if you can replicate it? Let me know if I can help with that. :)

I think I'll add a helper function to help users add the appropriate text to their .Renviron

njtierney commented 3 years ago

I've got a PR that should introduce this system now, if anyone wants to take it for a whirl, you can install it with:

remotes::install_github("greta-dev/greta#359")
flyaflya commented 3 years ago

Thx for this. I think it needs a little tweaking. I gave it a go on a loaner MAC book that I have been using for testing. Failed at the install stage (see below)

> remotes::install_github("greta-dev/greta#359")
Downloading GitHub repo njtierney/greta@njt-config-reticulate-357
These packages have more recent versions available.
It is recommended to update all of them.
Which would you like to update?

1: All                               
2: CRAN packages only                
3: None                              
4: processx   (3.4.5 -> 3.5.0) [CRAN]
5: tensorflow (2.2.0 -> 2.4.0) [CRAN]

Enter one or more numbers, or an empty line to skip updates:1
processx   (3.4.5 -> 3.5.0) [CRAN]
tensorflow (2.2.0 -> 2.4.0) [CRAN]
Installing 2 packages: processx, tensorflow
trying URL 'https://cran.rstudio.com/bin/macosx/contrib/4.0/processx_3.5.0.tgz'
Content type 'application/x-gzip' length 291449 bytes (284 KB)
==================================================
downloaded 284 KB

trying URL 'https://cran.rstudio.com/bin/macosx/contrib/4.0/tensorflow_2.4.0.tgz'
Content type 'application/x-gzip' length 146103 bytes (142 KB)
==================================================
downloaded 142 KB

The downloaded binary packages are in
    /var/folders/f4/p63fbl2s4k9f_v70jmj29xlc0000gp/T//RtmpMpqVD1/downloaded_packages
✓  checking for file ‘/private/var/folders/f4/p63fbl2s4k9f_v70jmj29xlc0000gp/T/RtmpMpqVD1/remotes4a160d31beb/njtierney-greta-9291ce1/DESCRIPTION’ ...
─  preparing ‘greta’: (375ms)
✓  checking DESCRIPTION meta-information ...
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
─  building ‘greta_0.3.1.9012.tar.gz’

* installing *source* package ‘greta’ ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘greta’:
 .onLoad failed in loadNamespace() for 'greta', details:
  call: py_call_impl(callable, dots$args, dots$keywords)
  error: AttributeError: 'module' object has no attribute '__import__'

Detailed traceback: 
  File "/Library/Frameworks/R.framework/Versions/4.0/Resources/library/reticulate/python/rpytools/loader.py", line 13, in initialize
    _import = builtins.__import__

Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/Library/Frameworks/R.framework/Versions/4.0/Resources/library/greta’
Error: Failed to install 'greta' from GitHub:
  (converted from warning) installation of package ‘/var/folders/f4/p63fbl2s4k9f_v70jmj29xlc0000gp/T//RtmpMpqVD1/file4a16f8134ed/greta_0.3.1.9012.tar.gz’ had non-zero exit status
njtierney commented 3 years ago

OK thank you very much, @flyaflya for testing that out!

One question - of the python packages that you describe here:

reticulate::conda_install(envname = "r-reticulate",
                          packages =
                            c(
                              "python=3.7",
                              "tensorflow=1.14",
                              "pyyaml",
                              "requests",
                              "Pillow",
                              "pip",
                              "numpy=1.16",
                              "h5py=2.8",
                              "tensorflow-probability=0.7"
                            ))

are any of these needed for Causact? Or are these for greta? I couldn't find these being needed in Causact but just wanted to double check.

njtierney commented 3 years ago

It looks like we can't use this reticulate approach as we need to control the version of python used, and this isn't currently supported in reticulate.

Thanks to this issue though I've been able to take a much closer look at how installation works within greta and the python world, and I think we've been able to get quite close to solving this now! :)

If you are able to give the current master branch on github a spin, there is now an interactive approach to installing python and the python dependencies, let me know if there are any issues with installation and I'll happily take a look see :)

flyaflya commented 3 years ago

Tested it out on a MAC and the installation will need some work still. Thanks for tackling this though!!

The first install attempt failed to load cli when I requested a binary install be used. I then re-ran with compile from source and got further, but then got this error during installation of greta:

** testing if installed package can be loaded from temporary location
Error: package or namespace load failed for ‘greta’:
 .onLoad failed in loadNamespace() for 'greta', details:
  call: NULL
  error: Unable to find conda binary. Is Anaconda installed?
Error: loading failed
Execution halted
ERROR: loading failed
* removing ‘/Library/Frameworks/R.framework/Versions/4.1/Resources/library/greta’
Warning message:
In i.p(...) :
  installation of package ‘/var/folders/f4/p63fbl2s4k9f_v70jmj29xlc0000gp/T//RtmpNowhcF/file6b96f9e0e28/greta_0.3.1.9012.tar.gz’ had non-zero exit status

So I then installed reticulate and ran reticulate::install_miniconda(). This then allowed the greta installation to appear successful. However, I then ran into this:

> library("greta")
Error: package or namespace load failed for ‘greta’:
 .onLoad failed in loadNamespace() for 'tensorflow', details:
  call: py_module_import(module, convert = convert)
  error: ModuleNotFoundError: No module named 'tensorflow'

Here is my session info if it helps:

> sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 11.2.3

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] reticulate_1.20

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.6        compiler_4.1.0    prettyunits_1.1.1 base64enc_0.1-3  
 [5] remotes_2.4.0     tools_4.1.0       progress_1.2.2    digest_0.6.27    
 [9] jsonlite_1.7.2    lifecycle_1.0.0   lattice_0.20-44   pkgconfig_2.0.3  
[13] png_0.1-7         rlang_0.4.11      Matrix_1.3-3      cli_3.0.0        
[17] parallel_4.1.0    coda_0.19-4       vctrs_0.3.8       globals_0.14.0   
[21] rappdirs_0.3.3    hms_1.1.0         grid_4.1.0        listenv_0.8.0    
[25] R6_2.5.0          processx_3.5.2    parallelly_1.26.1 callr_3.7.0      
[29] whisker_0.4       magrittr_2.0.1    codetools_0.2-18  ps_1.6.0         
[33] ellipsis_0.3.2    tfruns_1.5.0      future_1.21.0     crayon_1.4.1

Let me know what else I can try or when you want me to test again. Thanks!!

flyaflya commented 3 years ago

also, just in case you overlooked this (I do not think you did), conda_install() can install a particular Python version:

reticulate::conda_install(envname = "r-reticulate",
                          packages =
                            c(
                              "python=3.7")
njtierney commented 3 years ago

Thanks so much, @flyaflya for testing this out - really appreciate it.

We need the version of 3.0.0 for {cli} at the moment, but hopefully that should be further clarified in #425

It looks like there might be an issue with how .onLoad is run when an R package is installed, and this is causing the package to error, as it tries to look for a special "greta-env" conda environment.

I'm working on resolving this now, and will ping you here when I've got some fixes in master.

RE the Python version - what I meant to say was that reticulate doesn't support using a python version in Config/reticulate.

Thanks again for helping us out 😄

njtierney commented 3 years ago

Hi @flyaflya - I've updated the master branch of greta with some changes that I think have addressed this after testing on docker - are you able to give it a go again?

flyaflya commented 3 years ago

The installation went smooth this time, but it still does not get me to the finish line. I run into a problem running install_tensorflow(). See below:

> library("greta")
> x = normal(0,10)
Error: 

This version of greta requires TensorFlow v1.14.0 and TensorFlow Probability v0.7.0, but TensorFlow isn't installed and TensorFlow Probability isn't installed. To install the correct versions do:

  install_tensorflow(
    version = "1.14.0",
    extra_packages = "tensorflow-probability"
  )
> install_tensorflow(
+   version = "1.14.0",
+   extra_packages = "tensorflow-probability"
+ )
Error: could not find a Python environment for /usr/bin/python

Here is the output of py_config() if it helps:

> reticulate::py_config()
python:         /usr/bin/python
libpython:      /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib
pythonhome:     /System/Library/Frameworks/Python.framework/Versions/2.7:/System/Library/Frameworks/Python.framework/Versions/2.7
version:        2.7.16 (default, Dec 21 2020, 23:00:36)  [GCC Apple LLVM 12.0.0 (clang-1200.0.30.4) [+internal-os, ptrauth-isa=sign+stri
numpy:          /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy
numpy_version:  1.8.0
tensorflow:     [NOT FOUND]

python versions found: 
 /usr/bin/python3
 /usr/bin/python

If I run reticulate::install_miniconda(), and then run install_tensorflow(), it still yields the same error message. Let me know your thoughts and I am happy to test again when needed. Thanks.

njtierney commented 3 years ago

Hi @flyaflya - interesting, the result of 

remotes::install_github("greta-dev/greta")

Should have resulted in a different event when running:

library(greta)

The process from start to finish should look like this:

Install current master branch of greta

# install.packages("remotes")
remotes::install_github("greta-dev/greta")

Load greta

library(greta)

image

Create a greta model

This will initialise python and trigger internal checks that make sure packages are installed. Something like this code is short and sweet and should trigger this.

model(normal(0,1))

image

Then this:

image

Follow these instructions:

Install greta dependencies

install_greta_deps()

image

image

image

Restart R + run library(greta)

library(greta)

Create a greta model

model(normal(0,1))

image

image

Let us know if this works! 😄

flyaflya commented 3 years ago

Thx for the detailed pics about what to expect. Not sure what I did wrong that led me to the install_tensorflow() message previously, but this time I get the install_greta_deps() message as expected. This gets me closer, but not quite there.

1). When I run install_greta_deps(), it installs minconda, but the RStudio console hangs indefinitely. I have to manually interrupt using the stop sign. Miniconda does install successfully though.

(base) lernerguest@BUEC-2RRFVH8 ~ % conda env list
# conda environments:
#
base                  *  /Users/lernerguest/Library/r-miniconda

2) So I manually interrupt it, restart R, and then run install_greta_deps() again. Again the console hangs indefinitely (this time after the console shows Installing python packages into 'greta-env' conda environment). This gets me closer, but does not get me Tensorflow as shown from the terminal:

(base) lernerguest@BUEC-2RRFVH8 ~ % conda activate greta-env
(greta-env) lernerguest@BUEC-2RRFVH8 ~ % conda list
# packages in environment at /Users/lernerguest/Library/r-miniconda/envs/greta-env:
#
# Name                    Version                   Build  Channel
ca-certificates           2021.5.30            h033912b_0    conda-forge
certifi                   2021.5.30        py37hf985489_0    conda-forge
libcxx                    11.1.0               habf9029_0    conda-forge
libffi                    3.3                  h046ec9c_2    conda-forge
ncurses                   6.2                  h2e338ed_4    conda-forge
openssl                   1.1.1k               h0d85af4_0    conda-forge
pip                       21.1.3             pyhd8ed1ab_0    conda-forge
python                    3.7.10          h7728216_100_cpython    conda-forge
python_abi                3.7                     2_cp37m    conda-forge
readline                  8.1                  h05e3726_0    conda-forge
setuptools                49.6.0           py37hf985489_3    conda-forge
sqlite                    3.36.0               h23a322b_0    conda-forge
tk                        8.6.10               h0419947_1    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xz                        5.2.5                haf1e3a3_1    conda-forge
zlib                      1.2.11            h7795811_1010    conda-forge

3). Restarting and reinstalling a third time with run install_greta_deps() makes zero change from step 2.

Below is the console output associated with the above.

R version 4.1.0 (2021-05-18) -- "Camp Pontanezen"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin17.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> remotes::install_github("greta-dev/greta")
Downloading GitHub repo greta-dev/greta@HEAD
Installing 1 packages: reticulate
trying URL 'https://cran.rstudio.com/bin/macosx/contrib/4.1/reticulate_1.20.tgz'
Content type 'application/x-gzip' length 1732726 bytes (1.7 MB)
==================================================
downloaded 1.7 MB

The downloaded binary packages are in
    /var/folders/f4/p63fbl2s4k9f_v70jmj29xlc0000gp/T//RtmpNmrcQq/downloaded_packages
Running `R CMD build`...
* checking for file ‘/private/var/folders/f4/p63fbl2s4k9f_v70jmj29xlc0000gp/T/RtmpNmrcQq/remotes45f311e15f3/greta-dev-greta-3b128d6/DESCRIPTION’ ... OK
* preparing ‘greta’:
* checking DESCRIPTION meta-information ... OK
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
Omitted ‘LazyData’ from DESCRIPTION
* building ‘greta_0.3.1.9012.tar.gz’
* installing *source* package ‘greta’ ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (greta)
> library("greta")

Attaching package: ‘greta’

The following objects are masked from ‘package:stats’:

    binomial, cov2cor, poisson

The following objects are masked from ‘package:base’:

    %*%, apply, backsolve, beta, chol2inv, colMeans, colSums, diag, eigen,
    forwardsolve, gamma, identity, rowMeans, rowSums, sweep, tapply

> model(normal(0,1))
x Initialising python and checking dependencies ... failed
Error: We have detected that you do not have the expected python packages setup. You can set these up using:

    `install_greta_deps()`

Then, restart R and run:

    `library(greta)`

(Note: Your R session should not have initialised Tensorflow yet).
For more information, see `?install_greta_deps` 
> install_greta_deps()
x No miniconda detected, installing miniconda ... failed
> ## I interupted after 10 minutes of no progress

Restarting R session...

> model(normal(0,1))
Error in model(normal(0, 1)) : could not find function "model"
> library("greta")

Attaching package: ‘greta’

The following objects are masked from ‘package:stats’:

    binomial, cov2cor, poisson

The following objects are masked from ‘package:base’:

    %*%, apply, backsolve, beta, chol2inv, colMeans, colSums, diag, eigen,
    forwardsolve, gamma, identity, rowMeans, rowSums, sweep, tapply

> model(normal(0,1))
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /Users/lernerguest/Library/r-miniconda/envs/r-reticulate

  added / updated specs:
    - numpy
    - python=3.6

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2021.5.30  |       h033912b_0         136 KB  conda-forge
    certifi-2021.5.30          |   py36h79c6626_0         141 KB  conda-forge
    libblas-3.9.0              |       9_openblas          11 KB  conda-forge
    libcblas-3.9.0             |       9_openblas          11 KB  conda-forge
    libcxx-11.1.0              |       habf9029_0         1.0 MB  conda-forge
    libffi-3.3                 |       h046ec9c_2          45 KB  conda-forge
    libgfortran-5.0.0          |9_3_0_h6c81a4c_22          19 KB  conda-forge
    libgfortran5-9.3.0         |      h6c81a4c_22         1.7 MB  conda-forge
    liblapack-3.9.0            |       9_openblas          11 KB  conda-forge
    libopenblas-0.3.15         |openmp_h5e1b9a4_1         8.7 MB  conda-forge
    llvm-openmp-11.1.0         |       hda6cdc1_1         268 KB  conda-forge
    ncurses-6.2                |       h2e338ed_4         881 KB  conda-forge
    numpy-1.19.5               |   py36h08dc641_1         5.0 MB  conda-forge
    openssl-1.1.1k             |       h0d85af4_0         1.9 MB  conda-forge
    pip-21.1.3                 |     pyhd8ed1ab_0         1.1 MB  conda-forge
    python-3.6.13              |h7728216_0_cpython        20.6 MB  conda-forge
    python_abi-3.6             |          2_cp36m           4 KB  conda-forge
    readline-8.1               |       h05e3726_0         266 KB  conda-forge
    setuptools-49.6.0          |   py36h79c6626_3         916 KB  conda-forge
    sqlite-3.36.0              |       h23a322b_0         1.7 MB  conda-forge
    tk-8.6.10                  |       h0419947_1         3.3 MB  conda-forge
    wheel-0.36.2               |     pyhd3deb0d_0          31 KB  conda-forge
    xz-5.2.5                   |       haf1e3a3_1         228 KB  conda-forge
    zlib-1.2.11                |    h7795811_1010         102 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        48.0 MB

The following NEW packages will be INSTALLED:

  ca-certificates    conda-forge/osx-64::ca-certificates-2021.5.30-h033912b_0
  certifi            conda-forge/osx-64::certifi-2021.5.30-py36h79c6626_0
  libblas            conda-forge/osx-64::libblas-3.9.0-9_openblas
  libcblas           conda-forge/osx-64::libcblas-3.9.0-9_openblas
  libcxx             conda-forge/osx-64::libcxx-11.1.0-habf9029_0
  libffi             conda-forge/osx-64::libffi-3.3-h046ec9c_2
  libgfortran        conda-forge/osx-64::libgfortran-5.0.0-9_3_0_h6c81a4c_22
  libgfortran5       conda-forge/osx-64::libgfortran5-9.3.0-h6c81a4c_22
  liblapack          conda-forge/osx-64::liblapack-3.9.0-9_openblas
  libopenblas        conda-forge/osx-64::libopenblas-0.3.15-openmp_h5e1b9a4_1
  llvm-openmp        conda-forge/osx-64::llvm-openmp-11.1.0-hda6cdc1_1
  ncurses            conda-forge/osx-64::ncurses-6.2-h2e338ed_4
  numpy              conda-forge/osx-64::numpy-1.19.5-py36h08dc641_1
  openssl            conda-forge/osx-64::openssl-1.1.1k-h0d85af4_0
  pip                conda-forge/noarch::pip-21.1.3-pyhd8ed1ab_0
  python             conda-forge/osx-64::python-3.6.13-h7728216_0_cpython
  python_abi         conda-forge/osx-64::python_abi-3.6-2_cp36m
  readline           conda-forge/osx-64::readline-8.1-h05e3726_0
  setuptools         conda-forge/osx-64::setuptools-49.6.0-py36h79c6626_3
  sqlite             conda-forge/osx-64::sqlite-3.36.0-h23a322b_0
  tk                 conda-forge/osx-64::tk-8.6.10-h0419947_1
  wheel              conda-forge/noarch::wheel-0.36.2-pyhd3deb0d_0
  xz                 conda-forge/osx-64::xz-5.2.5-haf1e3a3_1
  zlib               conda-forge/osx-64::zlib-1.2.11-h7795811_1010

Downloading and Extracting Packages
llvm-openmp-11.1.0   | 268 KB    | ########## | 100% 
xz-5.2.5             | 228 KB    | ########## | 100% 
libcblas-3.9.0       | 11 KB     | ########## | 100% 
libblas-3.9.0        | 11 KB     | ########## | 100% 
pip-21.1.3           | 1.1 MB    | ########## | 100% 
ncurses-6.2          | 881 KB    | ########## | 100% 
certifi-2021.5.30    | 141 KB    | ########## | 100% 
openssl-1.1.1k       | 1.9 MB    | ########## | 100% 
wheel-0.36.2         | 31 KB     | ########## | 100% 
ca-certificates-2021 | 136 KB    | ########## | 100% 
libgfortran-5.0.0    | 19 KB     | ########## | 100% 
libcxx-11.1.0        | 1.0 MB    | ########## | 100% 
liblapack-3.9.0      | 11 KB     | ########## | 100% 
zlib-1.2.11          | 102 KB    | ########## | 100% 
numpy-1.19.5         | 5.0 MB    | ########## | 100% 
setuptools-49.6.0    | 916 KB    | ########## | 100% 
libopenblas-0.3.15   | 8.7 MB    | ########## | 100% 
python-3.6.13        | 20.6 MB   | ########## | 100% 
python_abi-3.6       | 4 KB      | ########## | 100% 
tk-8.6.10            | 3.3 MB    | ########## | 100% 
sqlite-3.36.0        | 1.7 MB    | ########## | 100% 
libgfortran5-9.3.0   | 1.7 MB    | ########## | 100% 
readline-8.1         | 266 KB    | ########## | 100% 
libffi-3.3           | 45 KB     | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda

#
# To activate this environment, use
#
#     $ conda activate /Users/lernerguest/Library/r-miniconda/envs/r-reticulate
#
# To deactivate an active environment, use
#
#     $ conda deactivate

x Initialising python and checking dependencies ... failed
Error: We have detected that you do not have the expected python packages setup. You can set these up using:

    `install_greta_deps()`

Then, restart R and run:

    `library(greta)`

(Note: Your R session should not have initialised Tensorflow yet).
For more information, see `?install_greta_deps` 
> install_greta_deps()  ## this can take several minutes 
✓ Creating 'greta-env' conda environment using python v3.7 ... done
• To see full installation notes run:
• `greta_notes_conda_create()`
x Installing python packages into 'greta-env' conda environment ... failed
> ## interupted again

R version 4.1.0 (2021-05-18) -- "Camp Pontanezen"
Copyright (C) 2021 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin17.0 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> remotes::install_github("greta-dev/greta")
Downloading GitHub repo greta-dev/greta@HEAD
Installing 1 packages: reticulate
trying URL 'https://cran.rstudio.com/bin/macosx/contrib/4.1/reticulate_1.20.tgz'
Content type 'application/x-gzip' length 1732726 bytes (1.7 MB)
==================================================
downloaded 1.7 MB

The downloaded binary packages are in
    /var/folders/f4/p63fbl2s4k9f_v70jmj29xlc0000gp/T//RtmpNmrcQq/downloaded_packages
Running `R CMD build`...
* checking for file ‘/private/var/folders/f4/p63fbl2s4k9f_v70jmj29xlc0000gp/T/RtmpNmrcQq/remotes45f311e15f3/greta-dev-greta-3b128d6/DESCRIPTION’ ... OK
* preparing ‘greta’:
* checking DESCRIPTION meta-information ... OK
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
Omitted ‘LazyData’ from DESCRIPTION
* building ‘greta_0.3.1.9012.tar.gz’
* installing *source* package ‘greta’ ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (greta)
> library("greta")

Attaching package: ‘greta’

The following objects are masked from ‘package:stats’:

    binomial, cov2cor, poisson

The following objects are masked from ‘package:base’:

    %*%, apply, backsolve, beta, chol2inv, colMeans, colSums, diag, eigen,
    forwardsolve, gamma, identity, rowMeans, rowSums, sweep, tapply

> model(normal(0,1))
x Initialising python and checking dependencies ... failed
Error: We have detected that you do not have the expected python packages setup. You can set these up using:

    `install_greta_deps()`

Then, restart R and run:

    `library(greta)`

(Note: Your R session should not have initialised Tensorflow yet).
For more information, see `?install_greta_deps` 
> install_greta_deps()
x No miniconda detected, installing miniconda ... failed
> ## I interupted after 10 minutes of no progress

Restarting R session...

> model(normal(0,1))
Error in model(normal(0, 1)) : could not find function "model"
> library("greta")

Attaching package: ‘greta’

The following objects are masked from ‘package:stats’:

    binomial, cov2cor, poisson

The following objects are masked from ‘package:base’:

    %*%, apply, backsolve, beta, chol2inv, colMeans, colSums, diag, eigen,
    forwardsolve, gamma, identity, rowMeans, rowSums, sweep, tapply

> model(normal(0,1))
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /Users/lernerguest/Library/r-miniconda/envs/r-reticulate

  added / updated specs:
    - numpy
    - python=3.6

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2021.5.30  |       h033912b_0         136 KB  conda-forge
    certifi-2021.5.30          |   py36h79c6626_0         141 KB  conda-forge
    libblas-3.9.0              |       9_openblas          11 KB  conda-forge
    libcblas-3.9.0             |       9_openblas          11 KB  conda-forge
    libcxx-11.1.0              |       habf9029_0         1.0 MB  conda-forge
    libffi-3.3                 |       h046ec9c_2          45 KB  conda-forge
    libgfortran-5.0.0          |9_3_0_h6c81a4c_22          19 KB  conda-forge
    libgfortran5-9.3.0         |      h6c81a4c_22         1.7 MB  conda-forge
    liblapack-3.9.0            |       9_openblas          11 KB  conda-forge
    libopenblas-0.3.15         |openmp_h5e1b9a4_1         8.7 MB  conda-forge
    llvm-openmp-11.1.0         |       hda6cdc1_1         268 KB  conda-forge
    ncurses-6.2                |       h2e338ed_4         881 KB  conda-forge
    numpy-1.19.5               |   py36h08dc641_1         5.0 MB  conda-forge
    openssl-1.1.1k             |       h0d85af4_0         1.9 MB  conda-forge
    pip-21.1.3                 |     pyhd8ed1ab_0         1.1 MB  conda-forge
    python-3.6.13              |h7728216_0_cpython        20.6 MB  conda-forge
    python_abi-3.6             |          2_cp36m           4 KB  conda-forge
    readline-8.1               |       h05e3726_0         266 KB  conda-forge
    setuptools-49.6.0          |   py36h79c6626_3         916 KB  conda-forge
    sqlite-3.36.0              |       h23a322b_0         1.7 MB  conda-forge
    tk-8.6.10                  |       h0419947_1         3.3 MB  conda-forge
    wheel-0.36.2               |     pyhd3deb0d_0          31 KB  conda-forge
    xz-5.2.5                   |       haf1e3a3_1         228 KB  conda-forge
    zlib-1.2.11                |    h7795811_1010         102 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        48.0 MB

The following NEW packages will be INSTALLED:

  ca-certificates    conda-forge/osx-64::ca-certificates-2021.5.30-h033912b_0
  certifi            conda-forge/osx-64::certifi-2021.5.30-py36h79c6626_0
  libblas            conda-forge/osx-64::libblas-3.9.0-9_openblas
  libcblas           conda-forge/osx-64::libcblas-3.9.0-9_openblas
  libcxx             conda-forge/osx-64::libcxx-11.1.0-habf9029_0
  libffi             conda-forge/osx-64::libffi-3.3-h046ec9c_2
  libgfortran        conda-forge/osx-64::libgfortran-5.0.0-9_3_0_h6c81a4c_22
  libgfortran5       conda-forge/osx-64::libgfortran5-9.3.0-h6c81a4c_22
  liblapack          conda-forge/osx-64::liblapack-3.9.0-9_openblas
  libopenblas        conda-forge/osx-64::libopenblas-0.3.15-openmp_h5e1b9a4_1
  llvm-openmp        conda-forge/osx-64::llvm-openmp-11.1.0-hda6cdc1_1
  ncurses            conda-forge/osx-64::ncurses-6.2-h2e338ed_4
  numpy              conda-forge/osx-64::numpy-1.19.5-py36h08dc641_1
  openssl            conda-forge/osx-64::openssl-1.1.1k-h0d85af4_0
  pip                conda-forge/noarch::pip-21.1.3-pyhd8ed1ab_0
  python             conda-forge/osx-64::python-3.6.13-h7728216_0_cpython
  python_abi         conda-forge/osx-64::python_abi-3.6-2_cp36m
  readline           conda-forge/osx-64::readline-8.1-h05e3726_0
  setuptools         conda-forge/osx-64::setuptools-49.6.0-py36h79c6626_3
  sqlite             conda-forge/osx-64::sqlite-3.36.0-h23a322b_0
  tk                 conda-forge/osx-64::tk-8.6.10-h0419947_1
  wheel              conda-forge/noarch::wheel-0.36.2-pyhd3deb0d_0
  xz                 conda-forge/osx-64::xz-5.2.5-haf1e3a3_1
  zlib               conda-forge/osx-64::zlib-1.2.11-h7795811_1010

Downloading and Extracting Packages
llvm-openmp-11.1.0   | 268 KB    | ########## | 100% 
xz-5.2.5             | 228 KB    | ########## | 100% 
libcblas-3.9.0       | 11 KB     | ########## | 100% 
libblas-3.9.0        | 11 KB     | ########## | 100% 
pip-21.1.3           | 1.1 MB    | ########## | 100% 
ncurses-6.2          | 881 KB    | ########## | 100% 
certifi-2021.5.30    | 141 KB    | ########## | 100% 
openssl-1.1.1k       | 1.9 MB    | ########## | 100% 
wheel-0.36.2         | 31 KB     | ########## | 100% 
ca-certificates-2021 | 136 KB    | ########## | 100% 
libgfortran-5.0.0    | 19 KB     | ########## | 100% 
libcxx-11.1.0        | 1.0 MB    | ########## | 100% 
liblapack-3.9.0      | 11 KB     | ########## | 100% 
zlib-1.2.11          | 102 KB    | ########## | 100% 
numpy-1.19.5         | 5.0 MB    | ########## | 100% 
setuptools-49.6.0    | 916 KB    | ########## | 100% 
libopenblas-0.3.15   | 8.7 MB    | ########## | 100% 
python-3.6.13        | 20.6 MB   | ########## | 100% 
python_abi-3.6       | 4 KB      | ########## | 100% 
tk-8.6.10            | 3.3 MB    | ########## | 100% 
sqlite-3.36.0        | 1.7 MB    | ########## | 100% 
libgfortran5-9.3.0   | 1.7 MB    | ########## | 100% 
readline-8.1         | 266 KB    | ########## | 100% 
libffi-3.3           | 45 KB     | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda

#
# To activate this environment, use
#
#     $ conda activate /Users/lernerguest/Library/r-miniconda/envs/r-reticulate
#
# To deactivate an active environment, use
#
#     $ conda deactivate

x Initialising python and checking dependencies ... failed
Error: We have detected that you do not have the expected python packages setup. You can set these up using:

    `install_greta_deps()`

Then, restart R and run:

    `library(greta)`

(Note: Your R session should not have initialised Tensorflow yet).
For more information, see `?install_greta_deps` 
> install_greta_deps()  ## this can take several minutes 
✓ Creating 'greta-env' conda environment using python v3.7 ... done
• To see full installation notes run:
• `greta_notes_conda_create()`
x Installing python packages into 'greta-env' conda environment ... failed
> ## interupted again

Restarting R session...

> library("greta")

Attaching package: ‘greta’

The following objects are masked from ‘package:stats’:

    binomial, cov2cor, poisson

The following objects are masked from ‘package:base’:

    %*%, apply, backsolve, beta, chol2inv, colMeans, colSums, diag, eigen,
    forwardsolve, gamma, identity, rowMeans, rowSums, sweep, tapply

> model(normal(0,1))
x Initialising python and checking dependencies ... failed
Error: We have detected that you do not have the expected python packages setup. You can set these up using:

    `install_greta_deps()`

Then, restart R and run:

    `library(greta)`

(Note: Your R session should not have initialised Tensorflow yet).
For more information, see `?install_greta_deps` 
> install_greta_deps()  ## this can take several minutes 
✓ Creating 'greta-env' conda environment using python v3.7 ... done
• To see full installation notes run:
• `greta_notes_conda_create()`
x Installing python packages into 'greta-env' conda environment ... failed

Here is my session info:

> sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 11.2.3

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] greta_0.3.1.9012

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.6        rstudioapi_0.13   parallelly_1.26.1 magrittr_2.0.1   
 [5] whisker_0.4       hms_1.1.0         progress_1.2.2    lattice_0.20-44  
 [9] R6_2.5.0          rlang_0.4.11      globals_0.14.0    tools_4.1.0      
[13] parallel_4.1.0    grid_4.1.0        png_0.1-7         cli_3.0.0        
[17] coda_0.19-4       tfruns_1.5.0      ellipsis_0.3.2    digest_0.6.27    
[21] lifecycle_1.0.0   crayon_1.4.1      tensorflow_2.5.0  processx_3.5.2   
[25] Matrix_1.3-3      callr_3.7.0       base64enc_0.1-3   vctrs_0.3.8      
[29] ps_1.6.0          codetools_0.2-18  glue_1.4.2        compiler_4.1.0   
[33] prettyunits_1.1.1 reticulate_1.20   jsonlite_1.7.2    future_1.21.0    
[37] listenv_0.8.0     pkgconfig_2.0.3
njtierney commented 3 years ago

Thanks for this - @flyaflya - thanks for testing this out again, sorry it didn't quite work out as smoothly as I was hoping.

When you say, "hangs indefinitely", how long was this?

I should change the message there to state that this might take a few minutes. I've experienced a lot of variability with this. Sometimes it is a 1-2 minutes, and other times it takes 5, 10, or even 15 minutes when I tried it on a docker container the other day.

I'm not sure why it's taking so long - ideally this is something that should happen quickly, it could be that some of the python installation steps are taking a while.

We currently hide all of the python code output as there is a lot of text that is output and drowns the console a bit - would it be helpful if we made a verbose = TRUE option, so you could see that things are happening?

flyaflya commented 3 years ago

Hangs indefinitely -- I re-tested this morning, and on my MAC I am still waiting for a response after 50 minutes. (note: I tested this on my Windows machine and it ran perfectly in under 4 minutes :-) ).

The stage it hangs at is Installing python packages in greta-env conda environment when I have miniconda already installed. And it hangs at No miniconda detected, installing miniconda when I do not have conda already.

Looking at the code, I ran reticulate::conda_install("greta-env" ... directly and that worked quickly on my MAC.

So it must be something in these lines (and corresponding miniconda install lines) that do not work right on a MAC:

  r_conda_create <- r_process$new(callr_conda_create)
  r_conda_create$wait()
  greta_stash$conda_create_notes <- r_conda_create$read_output()
  cli_process_done()

My guess is reticulate::install_miniconda and reticulate::conda_install() behave differently when run interactively versus using callr, but I cannot figure out why they both hang. If it helps, callr_install_miniconda is actually successful as miniconda does get installed whereas callr_conda_install is not successful as Tensorflow does not get installed. There is also greta's callr_conda_create that gets executed within install_greta_deps() and that one line does not hang indefinitely and seems to work.

njtierney commented 3 years ago

Hi @flyaflya - thanks so much for taking the time to do this, we really appreciate it!

Glad to hear that it ran perfectly on the windows machine - that is awesome news.

It's good to know that the reticulate::conda_install("greta-env" ... code works on your MAC - interesting that there might be an issue with callr and friends there.

To help as a workaround with this, we've introduced a new timeout argument, which is discussed in #431 - if it's alright with you, could we move the discussion on this topic to that issue and I close this issue thread?

This is now in master, so you should be able to get the new installation help at

remotes::install_github("greta-dev/greta")
njtierney commented 3 years ago

Ack, @flyaflya I've found a bug, so just hold off from testing this for the moment.

flyaflya commented 3 years ago

I am happy to keep testing when you say all is well. Just as a heads up, I am not confident the timeout argument will work for the set-up of the greta-env as the callr_conda_install section of the code does not seem to lead to a successful install of Tensorflow. The R session must hang prior to actually installing that package in the python environment.

I subscribed to notfications on the other thread, so happy to move there if you want.

njtierney commented 2 years ago

Hi @flyaflya and other folks here!

I've resolved #431 with #491 - I'm hopeful that this should resolve some of these issues with the timeout etc, if you want to give it a go, it's merged into master now.

flyaflya commented 2 years ago

Thanks @njtierney !!! I have just tested out the DEV install process on several Windows and Mac (non-m1) machines. Seems to be working really well. Well done!

njtierney commented 2 years ago

I am so very happy to hear that it works! It was hard to test. Nearly on CRAN now, just got to iron out some strange test behaviours.

On Fri, 11 Mar 2022, 00:14 Adam Fleischhacker, @.***> wrote:

Thanks @njtierney https://github.com/njtierney !!! I have just tested out the DEV install process on several Windows and Mac (non-m1) machines. Seems to be working really well. Well done!

— Reply to this email directly, view it on GitHub https://github.com/greta-dev/greta/issues/357#issuecomment-1064232890, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRQDJLGHALR24ODVI2R6MLU7INVVANCNFSM4S6CZSWQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>