Error with install_keras() from behind a firewall #rstudio #keras

[d3pl0y]

From @d3pl0y on January 4, 2018 13:34

Hi, pulling my hairs out after a few attempts at having keras on my corporate windows laptop. Long story short, the "simple" installation never worked because it has an HTTP error

>install.packages("keras")
>library(keras)
>install_keras()

Using r-tensorflow conda environment for TensorFlow installation
 Fetching package metadata ...
 CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/conda-forge/win-64/repodata.json>
 Elapsed: -

 An HTTP error occurred when trying to retrieve this URL.
 HTTP errors are often intermittent, and a simple retry will get you on your way.
 ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='conda.anaconda.org', port=443): Max retries exceeded with url: /conda-forge/win-64/repodata.json (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x0000000004E62470>, 'Connection to conda.anaconda.org timed out. (connect timeout=9.15)'))",),)

 Error: Error 1 occurred installing packages into conda environment r-tensorflow
 In addition: Warning message:
 running command '"D:\DATA\user\APPLIC~1\ANACON~1\Scripts\conda.exe" "install" "-c" "conda-forge" "--yes" "--name" "r-tensorflow" "tensorflow" "h5py" "pyyaml" "requests" "Pillow" "keras" "scipy"' had status 1 

Fair enough, following this link I edited the .condarc file as follows (there is just one - before "defaults")

channels: -- defaults proxy_servers: http: http://user:pass@proxy1:port https: https://user:pass@proxy2.18:port ssl_verify: False

and in a few attempts at least the Anaconda Prompt did not show the same HTTP connection error. Following issue #147 (thanks @jjallaire and @andreidi) I then, inside Anaconda Prompt ran

1. conda create --yes --name r-tensorflow 
2. activate r-tensorflow
3. conda install -c conda-forge keras

which asked to install a lot of new packages. At this point if I try the following I get a new error

>library(keras) 
>dataset_mnist()
Using TensorFlow backend.
Error: ModuleNotFoundError: No module named 'tensorflow'

I tried if I can load the tf but I still get

>library(tensorflow)
>dataset_mnist()
Using TensorFlow backend.
Error: Python module keras was not found.
Detected Python configuration:

python:         D:\DATA\user\Applications\anaconda3\envs\r-tensorflow\python.exe
libpython:      D:/DATA/user/Applications/anaconda3/envs/r-tensorflow/python36.dll
pythonhome:     D:\DATA\user\APPLIC~1\ANACON~1\envs\R-TENS~1
version:        3.6.4 | packaged by conda-forge | (default, Dec 24 2017, 10:11:43) [MSC v.1900 64 bit (AMD64)]
Architecture:   64bit
numpy:          D:\DATA\user\APPLIC~1\ANACON~1\envs\R-TENS~1\lib\site-packages\numpy
numpy_version:  1.13.3
keras:          D:\DATA\user\APPLIC~1\ANACON~1\envs\R-TENS~1\lib\site-packages\keras

python versions found: 
 D:\DATA\user\Applications\anaconda3\envs\r-tensorflow\python.exe
 D:\DATA\user\APPLIC~1\ANACON~1\python.exe

and if check the links with reticulate() I find the same output

> reticulate::py_config()
python:         D:\DATA\user\Applications\anaconda3\envs\r-tensorflow\python.exe
libpython:      D:/DATA/user/Applications/anaconda3/envs/r-tensorflow/python36.dll
pythonhome:     D:\DATA\user\APPLIC~1\ANACON~1\envs\R-TENS~1
version:        3.6.4 | packaged by conda-forge | (default, Dec 24 2017, 10:11:43) [MSC v.1900 64 bit (AMD64)]
Architecture:   64bit
numpy:          D:\DATA\user\APPLIC~1\ANACON~1\envs\R-TENS~1\lib\site-packages\numpy
numpy_version:  1.13.3
keras:          D:\DATA\user\APPLIC~1\ANACON~1\envs\R-TENS~1\lib\site-packages\keras

python versions found: 
 D:\DATA\user\Applications\anaconda3\envs\r-tensorflow\python.exe
 D:\DATA\user\APPLIC~1\ANACON~1\python.exe

problem is: pythonhome, numpy, keras and the second python versions point to a previous installation of Anaconda which I uninstalled yesterday while trying to solve the initials issue.

I am definitely out of my depth here, so bear with me also when it comes to definitions :) I have the feeling the solution is just around the corner but I don't know what to try anymore. Maybe change the paths somewhere/somehow (don't know how)?

If it helps, this is the sessionInfo()

> sessionInfo()
R version 3.4.3 (2017-11-30)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

Matrix products: default

locale:
[1] LC_COLLATE=German_Germany.1252  LC_CTYPE=German_Germany.1252   
[3] LC_MONETARY=German_Germany.1252 LC_NUMERIC=C                   
[5] LC_TIME=German_Germany.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] keras_2.1.2

loaded via a namespace (and not attached):
 [1] compiler_3.4.3   magrittr_1.5     R6_2.2.2         tools_3.4.3      whisker_0.3-2   
 [6] base64enc_0.1-3  yaml_2.1.16      Rcpp_0.12.14     reticulate_1.3.1 tensorflow_1.4.3
[11] zeallot_0.0.6    jsonlite_1.5     tfruns_1.1      

Copied from original issue: rstudio/reticulate#144

[d3pl0y]

Note. I did not disregard e.g. issue #95 but if I try

> library(tensorflow)
> sess = tf$Session()
Error: Installation of TensorFlow not found.

Python environments searched for 'tensorflow' package:
 D:\DATA\user\Applications\anaconda3\envs\r-tensorflow\python.exe
 D:\DATA\user\Applications\anaconda3\python.exe

You can install TensorFlow using the install_tensorflow() function.

Then I try the following but I am back to square one...

> install_tensorflow()
Using r-tensorflow conda environment for TensorFlow installation
Fetching package metadata ...
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/conda-forge/win-64/repodata.json>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='conda.anaconda.org', port=443): Max retries exceeded with url: /conda-forge/win-64/repodata.json (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x00000000046694A8>, 'Connection to conda.anaconda.org timed out. (connect timeout=9.15)'))",),)

Error: Error 1 occurred installing packages into conda environment r-tensorflow
In addition: Warning message:
running command '"D:\DATA\user\APPLIC~1\ANACON~1\envs\R-TENS~1\Scripts\conda.bat" "install" "-c" "conda-forge" "--yes" "--name" "r-tensorflow" "tensorflow" "h5py" "pyyaml" "requests" "Pillow" "scipy"' had status 1 

[jjallaire]

I'm not sure what's going on here, the HTTP errors are almost certainly related to conda not being able to traverse your corporate proxy server but I don't know enough about either conda or your proxy configuration to speculate or offer further suggestions.

You might want to try removing entirely your previous installation via reticulate::conda_remove() then attempting install_keras() again.

[d3pl0y]

Thanks, I tried

> library(reticulate)
> conda_remove()
Error in gsub("\"", "\\\\\"", string) : 
  argument "envname" is missing, with no default

with Sys.getenv() I don't see anything referring to conda/python/keras/tensorflow so I am not sure what I should do. Do I uninstall Anaconda and start from scratch?

[jjallaire]

conda_remove(“r-tensorflow”) On Thu, Jan 4, 2018 at 9:46 AM d3pl0y notifications@github.com wrote:

Thanks, I tried

library(reticulate) conda_remove() Error in gsub("\"", "\\\"", string) : argument "envname" is missing, with no default

with Sys.getenv() I don't see anything referring to conda/python/keras/tensorflow so I am not sure what I should do. Do I uninstall Anaconda and start from scratch?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rstudio/keras/issues/238#issuecomment-355300195, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGXx0rVEmhuKa1VC3ZwEvpF-we3CRB2ks5tHORHgaJpZM4RTDt1 .

[d3pl0y]

> conda_remove("r-tensorflow")

Package plan for package removal in environment D:\DATA\user\APPLIC~1\ANACON~1\envs\r-tensorflow:

The following packages will be REMOVED:

    certifi:        2017.11.5-py36_0      conda-forge
    h5py:           2.7.1-py36_2          conda-forge
    hdf5:           1.10.1-vc14_1         conda-forge [vc14]
    icc_rt:         2017.0.4-h97af966_0              
    intel-openmp:   2018.0.0-hd92c6cd_8              
    keras:          2.0.9-py36_0          conda-forge
    libgpuarray:    0.7.5-vc14_0          conda-forge [vc14]
    mako:           1.0.7-py36_0          conda-forge
    markupsafe:     1.0-py36_0            conda-forge
    mkl:            2018.0.1-h2108138_4              
    numpy:          1.13.3-py36ha320f96_0            
    pip:            9.0.1-py36_1          conda-forge
    pygpu:          0.7.5-py36_0          conda-forge
    python:         3.6.4-0               conda-forge
    pyyaml:         3.12-py36_1           conda-forge
    scipy:          1.0.0-py36h1260518_0             
    setuptools:     38.2.4-py36_0         conda-forge
    six:            1.11.0-py36_1         conda-forge
    theano:         0.9.0-py36_1          conda-forge
    vc:             14-0                  conda-forge
    vs2015_runtime: 14.0.25420-0          conda-forge
    wheel:          0.30.0-py_1           conda-forge
    wincertstore:   0.2-py36_0            conda-forge
    yaml:           0.1.7-vc14_0          conda-forge [vc14]
    zlib:           1.2.11-vc14_0         conda-forge [vc14]

and then, QED :(

> keras::install_keras()
Creating r-tensorflow conda environment for TensorFlow installation...
Fetching package metadata ...
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.continuum.io/pkgs/main/win-64/repodata.json.bz2>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='repo.continuum.io', port=443): Max retries exceeded with url: /pkgs/main/win-64/repodata.json.bz2 (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x0000000004568D68>, 'Connection to repo.continuum.io timed out. (connect timeout=9.15)'))",),)

Error: Error 1 occurred creating conda environment r-tensorflow
In addition: Warning message:
running command '"D:\DATA\user\APPLIC~1\ANACON~1\Scripts\conda.exe" "create" "--yes" "--name" "r-tensorflow" "python=3.6"' had status 1 

Incidentally, the environment and the Warning keep referencing something that does not exist. Note that in a browser https://repo.continuum.io/pkgs/main/win-64/ opens with no problems

[jjallaire]

I am not sure what's going on with your HTTP connection (and it's not something I can really debug effectively since it has to do with your system/environment)

[d3pl0y]

No worries @jjallaire and thanks for checking in anyways. I'd leave this open so if I find a way around my issue, I could update the thread.

[chidungkr]

Dear all,

when i type this command: model_keras <- keras_model_sequential()

I get this message from R: Error: Specified conda binary 'C:\Users\win10ls\ANACON~1/Scripts/conda.exe' does not exist.

Please show me how to handle this problem. Many thanks.

Here is my information about R environment:

R version 3.4.3 (2017-11-30) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale: [1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] tensorflow_1.5 keras_2.1.3

loaded via a namespace (and not attached): [1] Rcpp_0.12.15 zeallot_0.0.6 grid_3.4.3 plyr_1.8.4 R6_2.2.2
[6] jsonlite_1.5 gtable_0.2.0 magrittr_1.5 scales_0.5.0 ggplot2_2.2.1
[11] pillar_1.0.1 tfruns_1.1 rlang_0.1.6 lazyeval_0.2.0 whisker_0.3-2
[16] reticulate_1.4 tools_3.4.3 munsell_0.4.3 yaml_2.1.16 compiler_3.4.3
[21] base64enc_0.1-3 colorspace_1.3-2 tibble_1.4.1

[paramiyer]

@d3pl0y Any resolution on this., do you have any hairs left? Really I am splitting mine too. I have tried everything possible - but no luck yet

My proxy setting in conda works fine [conda update conda on conda prompt]it seems the issue is with calling conda via Rstudio for an update or installing conda related packages like tensorflow in Rstudio. As a last measure I have included conda in my env vars too, but still no go. If you have any update please share. I was thinking of raising a ticket in github/Rstudio

[d3pl0y]

@paramiyer no sorry, I resorted to crying and alcoholic beverages ;) and hopefully my company will have a stable analytics platform soon, so I should be able to just open a ticket and ask to add keras to the distro.

[paramiyer]

@d3pl0y I close to the same state :( I have opened a ticket with RStudio

https://github.com/rstudio/rstudio/issues/2222

Let us add more details there & keep experimenting. @jjallaire, if you have heard more on this please share

[whotank]

I have the same problem. Installed RStudio from Anaconda Navigator. Then in RStudio I get:

library(keras) install_keras() Creating r-tensorflow conda environment for TensorFlow installation... Solving environment: ...working... failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://repo.continuum.io/pkgs/main/win-64/repodata.json.bz2 Elapsed: -

An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. ConnectTimeout(MaxRetryError("HTTPSConnectionPool(host='repo.continuum.io', port=443): Max retries exceeded with url: /pkgs/main/win-64/repodata.json.bz2 (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x00000000055A2780>, 'Connection to repo.continuum.io timed out. (connect timeout=9.15)'))",),)

[2uanta]

@whotank what do you get when you click the link https://repo.continuum.io/pkgs/main/win-64/repodata.json.bz2 above? You should get to a page with what looks like garbage data. If you don't then you may want to debug that problem first.

[whotank]

@2uanta I do get garbage-looking data.

[image: Inline image 1]

On Fri, Feb 23, 2018 at 3:40 PM, Quan Nguyen notifications@github.com wrote:

@whotank https://github.com/whotank what do you get when you click the link https://repo.continuum.io/pkgs/main/win-64/repodata.json.bz2 above? You should get to a page with what looks like garbage data. If you don't then you may want to debug that problem first.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rstudio/keras/issues/238#issuecomment-368132287, or mute the thread https://github.com/notifications/unsubscribe-auth/AVTCmcMMykDnX_jeAO6J78ZwJoC6xddIks5tXyImgaJpZM4RTDt1 .

[2uanta]

@whotank which OS are you running? I am running Windows without problem with Anaconda 3-5.0.1. You may want to try to reinstall the latest version of conda. This may help: https://github.com/ContinuumIO/anaconda-issues/issues/2760

[paramiyer]

@2uanta I tried you recommendation & still not working. Could you provide a gist of what exactly needs to be done based on ContinuumIO/anaconda-issues#2760 I upgraded both conda & the navigator from conda prompt

[whotank]

I set my proxy and it worked.

[glitterzhang]

can you please show us how did you set your proxy ? @whotank

[o0oBluePhoenixo0o]

One thing I did to "by pass" the firewall is to add another folder to .libPaths() of R where you have full access to that folder (maybe that is your working shared drive).

Go to your R folder then "etc" folder, edit Rprofile.site with any text editor and add .First <- function(){ .libPaths("your path here") }

Close and run R again, test with .libPaths(), you will see your new folder as one of the library. Now install keras again and you would find no problem !