search

rstudio / keras3

R Interface to Keras
https://keras3.posit.co/
Other
831 stars 282 forks source link

Keras for R v2.1.6 failed to re-install under RStudio #976

Closed faltinl closed 4 years ago

faltinl commented 4 years ago

System information

Problem From Tensorflow 2.0.0 cpu, import error: DLL load failed #36138 I learned that my processor does not have AVX (Advanced Vector Extension) and is therefore unable to support Tensorflow 2.0. My HW limits the SW setup to tensorflow<1.6 and keras<2.2.0.

I have used Keras for R for the last 3 yrs under RStudio and recently tried to update my installation to level TF 2.0. This led to losing my working installation setup described above since attempts to restore the whole SW setup as it was from a back-up failed. However, trying to turn time back by re-installing everything as it was (i.e. R, RStudio, package versions & Anaconda3) did not succeed either.

When the standard conda installation sequence

>library(keras)                                      # 2.1.6      
>install_keras(method = "conda", tensorflow = "1.5") # 1.5

was completed, the start sequence for using Keras followed,

>library(reticulate)  # 1.10
>use_condaenv("r-tensorflow", required = TRUE)
>library(tensorflow)  # 1.5 
>library(keras)       # 2.1.6

producing the following:

General test results

> sessionInfo()
R version 3.5.2 (2018-12-20)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)
Matrix products: default
locale:
[1] LC_COLLATE=German_Germany.1252  LC_CTYPE=German_Germany.1252   
[3] LC_MONETARY=German_Germany.1252 LC_NUMERIC=C                   
[5] LC_TIME=German_Germany.1252    
attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     
other attached packages:
[1] keras_2.1.6     tensorflow_1.5  reticulate_1.10
loaded via a namespace (and not attached):
 [1] compiler_3.5.2  R6_2.4.1        magrittr_1.5    Matrix_1.2-15  
 [5] tools_3.5.2     whisker_0.4     base64enc_0.1-3 Rcpp_1.0.3     
 [9] grid_3.5.2      zeallot_0.1.0   jsonlite_1.6    tfruns_1.4     
[13] lattice_0.20-38
> tensorflow::tf_config()
TensorFlow v1.5.0 (C:\ANACON~1\envs\R-TENS~1\lib\site-packages\keras\__init__.p)
Python v3.6 (C:\Anaconda3\envs\r-tensorflow\python.exe)
> reticulate::py_config() 
python:         C:\Anaconda3\envs\r-tensorflow\python.exe
libpython:      C:/Anaconda3/envs/r-tensorflow/python36.dll
pythonhome:     C:\ANACON~1\envs\R-TENS~1
version:        3.6.10 |Anaconda, Inc.| (default, Jan  7 2020, 15:18:16) 
                    [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:           [NOT FOUND]
keras:          C:\ANACON~1\envs\R-TENS~1\lib\site-packages\keras\__init__.p
NOTE: Python version was forced by use_python function

Functional test for Tensorflow

> sess = tf$Session()
> hello <- tf$constant('Hello, TensorFlow!')
> sess$run(hello) # expected: b'Hello, TensorFlow!' 
b'Hello, TensorFlow!'

Tests for Keras

> is_keras_available()  
[1] FALSE
> implementation() 
Module(keras)
> keras.backend.backend()
Error in keras.backend.backend(): could not find function "keras.backend.backend"
> backend(convert = T)$backend()
Error: Python module keras was not found.
Detected Python configuration:
python:         C:\Anaconda3\envs\r-tensorflow\python.exe
libpython:      C:/Anaconda3/envs/r-tensorflow/python36.dll
pythonhome:     C:\ANACON~1\envs\R-TENS~1
version:        3.6.10 |Anaconda, Inc.| (default, Jan  7 2020, 15:18:16) 
                    [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:           [NOT FOUND]
keras:          C:\ANACON~1\envs\R-TENS~1\lib\site-packages\keras\__init__.p
NOTE: Python version was forced by use_python function

Conclusion Obviously, Tensorflow is working, but Keras (backend) is not available.

What I noticed during the installation runs: regardless of the version of Keras and the other two components (I tried more or less every combination admitted by various dependencies) from R perspective, among the list of newly installed Python packages invariably the packages 

...
keras:               2.3.1-py36h21ff451_0          conda-forge
keras-applications:  1.0.8-py_1                    conda-forge
keras-preprocessing: 1.1.0-py_0                    conda-forge
...

used to show up. And that cannot be correct because when I installed the original setup about 2 yrs ago, a Python version keras 2.3.1-py... most probably wasn't known yet. However, how can I find out more about this and where do I get a suitable previous version of this module using Anaconda (if this should really be the cause of the actual problem)?

Thank you in advance for any useful hints.

dfalbel commented 4 years ago

It seems to me that your installation is working fine.

is_keras_available is not the way to check if Keras is installed. It just checks if the keras python implementation is installed, but tensorflow also proivides a keras implementation.

keras.backend.backend() is not a function from the keras R package. You should use k_backend() for that.

faltinl commented 4 years ago

Sorry - I can' t follow. My primary aim is, of course, not to pass tests but to run otherwise proven ML programmes. The test, btw, is not passed, as the following two lines show:

> k_backend()   
 Error: ImportError: cannot import name 'tf_utils'

But, quite more important, a short programme produces identical error messages at each line containing keras-specific commands, like

...
> y2 <- iris$Species %>% as.numeric() %>% to_categorical() 
 Error: Python module keras was not found.
Detected Python configuration:
python:         C:\Anaconda3\envs\r-tensorflow\python.exe
libpython:      C:/Anaconda3/envs/r-tensorflow/python36.dll
pythonhome:     C:\ANACON~1\envs\R-TENS~1
version:        3.6.10 |Anaconda, Inc.| (default, Jan  7 2020, 15:18:16) 
                    [MSC v.1916 64 bit (AMD64)]
Architecture:   64bit
numpy:           [NOT FOUND]
keras:          C:\ANACON~1\envs\R-TENS~1\lib\site-packages\keras\__init__.p
NOTE: Python version was forced by use_python function 
...

and the same for cases like 

...
> input <- layer_input(shape = 3)
...
> dense <- layer_dense(input, units = 32)
...
> model <- 
+   keras_model(inputs = input, outputs = list(output1, output2))
... etc ...

and

> model %>% compile(  
+   optimizer = "adam",
+   loss = list(loss_mean_squared_error, loss_categorical_crossentropy)
+ )
 Error in eval(lhs, parent, parent) : object 'model' not found

So, much to my regret, I don't have the impression of an installation working fine...

dfalbel commented 4 years ago

Can you try installing older versions of the keras R package and the tensorflow package? We are no longer supporting TensorFlow v1.5 in the recent versions of the R package.

You can do this by running:

remotes::install_version("keras", version = "2.1.4", repos = "http://cran.us.r-project.org")
remotes::install_version("tensorflow", version = "1.5", repos = "http://cran.us.r-project.org")

You can also consider using a cloud based computing ennvironment that uses a more recent CPU. You can install rstudio server in any provider like AWS, google cloud or azure and then install tensorflow and keras. You could also try using TF in rstudio.cloud.

faltinl commented 4 years ago

Thank you for these helpful hints, which I really appreciate. I did as suggested and, in a first step, declined all requests for package updates (for tensorflow, reticulate, jsonlite, rlang & yaml). Thus Keras 2.1.4, Tensorflow 1.5., and (from my earlier installation, see above).

Results were exactly the same as in my first post above.

In a second go, I accepted all requests for updates, which led to tensorflow 2.0.0 and reticulate 1.14. There was a partial improvement insofar, as numpy has been found:

numpy:          C:/Anaconda3/envs/r-tensorflow/Lib/site-packages/numpy
numpy_version:  1.18.1

k_backend(), however still produced Error: ImportError: cannot import name 'tf_utils', as above and keras commands in the test program led to the same error messages as above, too.

Just as a random experiment I replaced keras 2.1.4 by keras 2.2.5 which surprisingly had keras commands in my test programm accepted down to layer definitions, but compile() and fit() again produced error messages like the following:

> model %>% compile(optimizer = "adam",  
    loss = list(loss_mean_squared_error, loss_categorical_crossentropy) )
Error in UseMethod("compile") : 
  no applicable method for 'compile' applied to an object of class
"c('keras._impl.keras.engine.training.Model', 
'keras._impl.keras.engine.topology.Network', 
'tensorflow.python.layers.network.GraphNetwork',
'keras._impl.keras.engine.topology.Layer', 
'tensorflow.python.layers.base.Layer', 
'python.builtin.object')"

Moreover something strange happend: 

> k_backend()    
 Error: Keras loaded from tensorflow v1.5, however version 1.9 is required. 
Please update with tensorflow::install_tensorflow().

And installing tensorflow 1.9 led to a repeated request to install tensorflow 2.0.0 again, thus completing a whole cycle of inconsistence. -

In the meantime I'm concluding that I have to accept the situation as it is: there is no way back to pre-TF2.0 Keras. I'll have to look for other approaches to continue my work with Keras for R - which I find really more practical than Python, I openly confess. Following your suggestions to use a suitable cloud based environment definitely is a viable alternative.

Fortunately, there seems to be an opportunity to use a much more recent computer than my present one. It has an Intel Core i7-9750H processor with instruction set extensions Intel SSE4.1, SSE4.2 & AVX2, which should permit to use the cpu-version of tensorflow 2.0.0 without any further tinkering around. And, additionally, there is a built-in NVIDIA GeForce RTX 2060 gpu, too, which I suppose should enable a further step forward for me. How do you think about that?

dfalbel commented 4 years ago

Yes, definitely using a new computer with a recent CPU (and GPU!) seems to me like the best option.