cannot run first example in "Deep learning with R"

[rolfmblindgren]

library(keras)

imdb <- dataset_imdb(num_words = 10000)

gives

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: Object arrays cannot be loaded when allow_pickle=False

Python 3.5 on Ubuntu 16.04

[JaneLiu0617]

got the same problem!!

[skeydan]

This is due to a change in numpy which made necessary a corresponding change in tensorflow:

https://github.com/tensorflow/tensorflow/commit/79a8d5cdad942b9853aa70b59441983b42a8aeb3

Assuming you are on the current CRAN release of 1.13.1, you have the following options:

1) install the current nightly instead (this already has the above fix), like so

install_tensorflow(version="nightly")

or

2) downgrade numpy to a version below 1.16.3, e.g.

pip uninstall numpy
pip install --upgrade numpy==1.16.1

This would have to be done in the env you use for TF from R (default: r-tensorflow).

[JaneLiu0617]

Thank you, it worked!

On Sun, Apr 28, 2019 at 23:26 Sigrid Keydana notifications@github.com wrote:

This is due to a change in numpy which made necessary a corresponding change in tensorflow:

tensorflow/tensorflow@79a8d5c https://github.com/tensorflow/tensorflow/commit/79a8d5cdad942b9853aa70b59441983b42a8aeb3

Assuming you are on the current CRAN release of 1.13.1, you have the following options:

1. install the current nightly instead (this already has the above fix), like so

install_tensorflow(version="nightly")

or

1. downgrade numpy to a version below 1.16.3, e.g.

pip uninstall numpy pip install --upgrade numpy==1.16.1

This would have to be done in the env you use for TF from R (default: r-tensorflow).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rstudio/keras/issues/765#issuecomment-487463524, or mute the thread https://github.com/notifications/unsubscribe-auth/AJB24463MNDEEKXWFF2Q7E3PS2ILFANCNFSM4HI65VPQ .

[mjg4243]

This worked for me, too. But further on in the code --

clf <- model_text_classifier(verbose=TRUE, glove_file="/home/mike/glove") %>% fit(x_train, y_train, time_limit=5*60)

gave me

Error in py_call_impl(callable, dots$args, dots$keywords) : TypeError: object of type 'int' has no len()

somewhere in a python call? Near as I can tell the class for the input is 'int'.

Any ideas?

And comment please on fact that with tensorflow(version="nightly") the gpu does not appear to be used.

Thanks.

[dannypfetzer]

Hi everybody,

sorry that I have to open the topic again, but I executed all of the suggested solutions above and they only worked for the imdb dataset. When I try to load the reuters dataset I get the same error...

reuters <- dataset_reuters(num_words = 10000) Error in py_call_impl(callable, dots$args, dots$keywords) : ValueError: Object arrays cannot be loaded when allow_pickle=False

Detailed traceback: File "C:\Users\DP\ANACON~2\envs\R-TENS~1\lib\site-packages\tensorflow\python\keras\datasets\reuters.py", line 84, in load_data xs, labels = f['x'], f['y'] File "C:\Users\DP\ANACON~2\envs\R-TENS~1\lib\site-packages\numpy\lib\npyio.py", line 262, in getitem pickle_kwargs=self.pickle_kwargs) File "C:\Users\DP\ANACON~2\envs\R-TENS~1\lib\site-packages\numpy\lib\format.py", line 696, in read_array raise ValueError("Object arrays cannot be loaded when "

Please help me with this problem. I can´t find anything on the web to help me solve it myself.

[skeydan]

Can you indicate your

tensorflow::tf_config()

and sessionInfo?

[dannypfetzer]

Thank you for the quick reply! Yes, of course. Here it is:

tensorflow::tf_config() TensorFlow v1.12.0 (C:\Users\DP\ANACON~2\envs\R-TENS~1\lib\site-packages\tensorflow__init__.p) Python v3.6 (C:\Users\DP\Anaconda3\envs\r-tensorflow\python.exe) sessionInfo function (package = NULL) { z <- list() z$R.version <- R.Version() z$platform <- z$R.version$platform if (nzchar(.Platform$r_arch)) z$platform <- paste(z$platform, .Platform$r_arch, sep = "/") z$platform <- paste0(z$platform, " (", 8 * .Machine$sizeof.pointer, "-bit)") z$locale <- Sys.getlocale() z$running <- osVersion z$RNGkind <- RNGkind() if (is.null(package)) { package <- grep("^package:", search(), value = TRUE) keep <- vapply(package, function(x) x == "package:base" || !is.null(attr(as.environment(x), "path")), NA) package <- .rmpkg(package[keep]) } pkgDesc <- lapply(package, packageDescription, encoding = NA) if (length(package) == 0) stop("no valid packages were specified") basePkgs <- sapply(pkgDesc, function(x) !is.null(x$Priority) && x$Priority == "base") z$basePkgs <- package[basePkgs] if (any(!basePkgs)) { z$otherPkgs <- pkgDesc[!basePkgs] names(z$otherPkgs) <- package[!basePkgs] } loadedOnly <- loadedNamespaces() loadedOnly <- loadedOnly[!(loadedOnly %in% package)] if (length(loadedOnly)) { names(loadedOnly) <- loadedOnly pkgDesc <- c(pkgDesc, lapply(loadedOnly, packageDescription)) z$loadedOnly <- pkgDesc[loadedOnly] } z$matprod <- as.character(options("matprod")) es <- extSoftVersion() z$BLAS <- as.character(es["BLAS"]) z$LAPACK <- La_library() class(z) <- "sessionInfo" z } <bytecode: 0x000000002dcecf98>

[mjg4243]

Sorry to disappoint you but delighted to hear from you. I'm out of town for a while and don't have access to the same machine. I will try again in a nonth or tow. Cheers Mike Gallagher

On Monday, June 3, 2019, 1:42:51 AM CDT, Sigrid Keydana <notifications@github.com> wrote:  

Can you indicate your

tensorflow::tf_config()

and sessionInfo?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[skeydan]

@mjg4243 thanks but this was not directed to you :-) @dannypfetzer as indicated above, this is fixed in the nightly build:

install_tensorflow(version="nightly")

[dannypfetzer]

Unfortunately it doesn´t work for me. To be sure I didn´t make a mistake I ran "install_tensorflow(version="nightly")" again and still get these results:

reuters <- dataset_reuters(num_words = 10000) Error in py_call_impl(callable, dots$args, dots$keywords) : ValueError: Object arrays cannot be loaded when allow_pickle=False

Detailed traceback: File "C:\Users\DP\ANACON~2\envs\R-TENS~1\lib\site-packages\tensorflow\python\keras\datasets\reuters.py", line 84, in load_data xs, labels = f['x'], f['y'] File "C:\Users\DP\ANACON~2\envs\R-TENS~1\lib\site-packages\numpy\lib\npyio.py", line 262, in getitem pickle_kwargs=self.pickle_kwargs) File "C:\Users\DP\ANACON~2\envs\R-TENS~1\lib\site-packages\numpy\lib\format.py", line 696, in read_array raise ValueError("Object arrays cannot be loaded when "

tf_config() TensorFlow v1.14.1-dev20190602 (C:\Users\DP\ANACON~2\envs\R-TENS~1\lib\site-packages\tensorflow__init__.p) Python v3.6 (C:\Users\DP\Anaconda3\envs\r-tensorflow\python.exe) sessionInfo() R version 3.6.0 (2019-04-26) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

Random number generation: RNG: Mersenne-Twister Normal: Inversion Sample: Rounding

locale: [1] LC_COLLATE=German_Germany.1252 LC_CTYPE=German_Germany.1252 LC_MONETARY=German_Germany.1252 LC_NUMERIC=C LC_TIME=German_Germany.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] zeallot_0.1.0 e1071_1.7-1 caret_6.0-84 lattice_0.20-38 reticulate_1.12 tensorflow_1.13.1 keras_2.2.4.1 openxlsx_4.1.0 data.table_1.12.2 [10] forcats_0.4.0 stringr_1.4.0 dplyr_0.8.0.1 purrr_0.3.2 readr_1.3.1 tidyr_0.8.3 tibble_2.1.1 ggplot2_3.1.1 tidyverse_1.2.1

loaded via a namespace (and not attached): [1] Rcpp_1.0.1 lubridate_1.7.4 class_7.3-15 assertthat_0.2.1 ipred_0.9-9 foreach_1.4.4 R6_2.4.0 cellranger_1.1.0
[9] plyr_1.8.4 backports_1.1.4 stats4_3.6.0 httr_1.4.0 pillar_1.3.1 tfruns_1.4 rlang_0.3.4 lazyeval_0.2.2
[17] readxl_1.3.1 rstudioapi_0.10 whisker_0.3-2 rpart_4.1-15 Matrix_1.2-17 splines_3.6.0 gower_0.2.0 munsell_0.5.0
[25] broom_0.5.2 compiler_3.6.0 modelr_0.1.4 pkgconfig_2.0.2 base64enc_0.1-3 nnet_7.3-12 tidyselect_0.2.5 prodlim_2018.04.18 [33] codetools_0.2-16 crayon_1.3.4 withr_2.1.2 ModelMetrics_1.2.2 MASS_7.3-51.4 recipes_0.1.5 grid_3.6.0 nlme_3.1-139
[41] jsonlite_1.6 gtable_0.3.0 magrittr_1.5 scales_1.0.0 zip_2.0.1 cli_1.1.0 stringi_1.4.3 reshape2_1.4.3
[49] timeDate_3043.102 xml2_1.2.0 generics_0.0.2 lava_1.6.5 iterators_1.0.10 tools_3.6.0 glue_1.3.1 hms_0.4.2
[57] survival_2.44-1.1 yaml_2.2.0 colorspace_1.4-1 rvest_0.3.3 haven_2.1.0

[skeydan]

@dannypfetzer you're right, evidently they didn't fix this for dataset_reuters() in TensorFlow... I'll open a PR there. What you can do though is downgrade numpy (in the r-tensorflow env):

pip uninstall numpy
pip install --upgrade numpy==1.16.1

[skeydan]

PR: https://github.com/tensorflow/tensorflow/pull/29440

closing this again, current workaround is downgrading `numpy'.

[dannypfetzer]

Thank you and I´m very sorry if I´m not getting something here, but I installed the nightly version of tensorflow, downgraded numpy to Version 1.16.1 and still get the same errormessages....

tf_config() TensorFlow v1.14.1-dev20190602 (C:\Users\DP\ANACON~2\envs\R-TENS~1\lib\site-packages\tensorflow__init__.p) Python v3.6 (C:\Users\DP\Anaconda3\envs\r-tensorflow\python.exe)

---

sessionInfo() R version 3.6.0 (2019-04-26) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

Random number generation: RNG: Mersenne-Twister Normal: Inversion Sample: Rounding

locale: [1] LC_COLLATE=German_Germany.1252 LC_CTYPE=German_Germany.1252 LC_MONETARY=German_Germany.1252 LC_NUMERIC=C
[5] LC_TIME=German_Germany.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] zeallot_0.1.0 e1071_1.7-1 caret_6.0-84 lattice_0.20-38 reticulate_1.12 tensorflow_1.13.1 keras_2.2.4.1 openxlsx_4.1.0
[9] data.table_1.12.2 forcats_0.4.0 stringr_1.4.0 dplyr_0.8.0.1 purrr_0.3.2 readr_1.3.1 tidyr_0.8.3 tibble_2.1.1
[17] ggplot2_3.1.1 tidyverse_1.2.1

loaded via a namespace (and not attached): [1] Rcpp_1.0.1 lubridate_1.7.4 class_7.3-15 assertthat_0.2.1 ipred_0.9-9 foreach_1.4.4 R6_2.4.0
[8] cellranger_1.1.0 plyr_1.8.4 backports_1.1.4 stats4_3.6.0 httr_1.4.0 pillar_1.3.1 tfruns_1.4
[15] rlang_0.3.4 lazyeval_0.2.2 readxl_1.3.1 rstudioapi_0.10 whisker_0.3-2 rpart_4.1-15 Matrix_1.2-17
[22] splines_3.6.0 gower_0.2.0 munsell_0.5.0 broom_0.5.2 compiler_3.6.0 modelr_0.1.4 pkgconfig_2.0.2
[29] base64enc_0.1-3 nnet_7.3-12 tidyselect_0.2.5 prodlim_2018.04.18 codetools_0.2-16 crayon_1.3.4 withr_2.1.2
[36] ModelMetrics_1.2.2 MASS_7.3-51.4 recipes_0.1.5 grid_3.6.0 nlme_3.1-139 jsonlite_1.6 gtable_0.3.0
[43] magrittr_1.5 scales_1.0.0 zip_2.0.1 cli_1.1.0 stringi_1.4.3 reshape2_1.4.3 timeDate_3043.102 [50] xml2_1.2.0 generics_0.0.2 lava_1.6.5 iterators_1.0.10 tools_3.6.0 glue_1.3.1 hms_0.4.2
[57] survival_2.44-1.1 yaml_2.2.0 colorspace_1.4-1 rvest_0.3.3 haven_2.1.0

[mickguy]

My virtual env was using version 1.12 not the installed version of 1.14. Running install_keras() upgraded tensorflow to 1.14 and fixed the problem.

[drag05]

I am an R user and I am going through "Deep Learning with R" book which I find quite interesting.

I am using NVidia GPU in Windows 10 environment. I went to the entire exercise of installing CUDA Tools with cuDNN dlls as result of following poorly, and sometimes contradictory instructions online but in the end I managed to run the MNIST and IMDB datasets on GPU.

I am a bit frustrated with bloating my computer with Anaconda "environments" and multiple versions of the same package which inexplicably MUST be DOWNGRADED to work.

I fail to see the point in downgrading package versions only to be able to read some data!

How can I trust this to work daily in a production environment if any package version sporadic update could create problems downstream?

The very construct seems poorly "stitched" for immediate need. The suggested solutions look more like patches to me.

I wonder if isn't more straightforward using R with NVIDIA Nsight Compute tool. Any suggestions are welcome!

[SomyKamble]

same problem with c(c(train_data,train_labels),c(test_data,test_labels)) %<-% dataset_reuters(num_words = 10000)

nothing worked for me

[drag05]

This issue is not solved; why is it closed?

[dfalbel]

@drag05 and @SomyKamble These errors were caused by a non backward compatible update of numpy that broke Keras some months ago.

If you installl the latest version of TensorFlow (2.0) the problem should be fixed.

Could you share what's your tensorflow::tf_config()?

[drag05]

Thank you for re-opening this issue! After updating tensorflow from anaconda prompt the tf_config shows the same version as before the update:

_TensorFlow v1.14.0 (C:\Anaconda3\envs\r-reticulate\lib\site-packages\tensorflow__init__.p) Python v3.6 (C:\Anaconda3\envs\r-reticulate\python.exe)

Is there any way of using Keras outside Python/Anaconda - maybe Keras for R directly?

[dfalbel]

How are you installing TensorFlow? Are you using the install_tensorflow() or directly from conda install?

Installing TensorFlow with conda install tensorflow is not officially supported and usually leads to hard to detect problems. You can install TensorFlow via pip inside a conda environment - that's what the install_tensorflow() function does.

[mjg4243]

I haven' t been using conda or dockers -- just download and then 'install'. I notice some reference to a 'reticulate' space, or something -- could it be that I need to specify a different directory or something? Thanks. Mike Gallagher

On Thursday, October 10, 2019, 10:47:10 PM CDT, Daniel Falbel <notifications@github.com> wrote:  

How are you installing TensorFlow? Are you using the install_tensorflow() or directly from conda install?

Installing TensorFlow with conda install tensorflow is not officially supported and usually leads to hard to detect problems. You can install TensorFlow via pip inside a conda environment - that's what the install_tensorflow() function does.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[mjg4243]

TensorFlow v1.13.1 (~/.local/lib/python3.6/site-packages/tensorflow) Python v3.6 (/usr/bin/python)

Mike Gallagher

On Thursday, October 10, 2019, 09:04:40 PM CDT, Daniel Falbel <notifications@github.com> wrote:  

@drag05 and @SomyKamble These errors were caused by a non backward compatible update of numpy that broke Keras some months ago.

If you installl the latest version of TensorFlow (2.0) the problem should be fixed.

Could you share what's your tensorflow::tf_config()?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[SomyKamble]

@dfalbel I would like to thank you and share my gratitude for replying and helping me out with the issue.

the issue has been resolved by using the following lines of code library(keras) install_keras(tensorflow = "2.0.0")

please keep the good work up 💯

[drag05]

Thanks to your suggestion, I have installed tensorflow from RStudio console:

>tensorflow::install_tensorflow(version = 'gpu')

and now, for

>tensorflow::tf_config()

I get

TensorFlow v2.0.0 (C:\Anaconda3\envs\r-reticulate\lib\site-packages\tensorflow__init__.p) Python v3.6 (C:\Anaconda3\envs\r-reticulate\python.exe)

However, none of my 3 Anaconda environments (base, r-reticulate and tensorflow) show the update. They keep showing tensorflow version 1.14.0 and tensorflow-gpu 1.14.0.

Actually, unlike the output above, the r-reticulate environment shows no tensorflow whatsoever althought it shows python.

I find this unnecessarily complicated and now I am not sure:

1. where the tensorflow-gpu v.2.0.0 resides.
2. whether I need the other older versions still installed

Suggestions are welcome, thank you!

[drag05]

@mjg4243 I assume environment r-reticulate is there because I use R which uses the R package reticulate to talk to python. I am looking for a R versions of keras/ tensorflow to eliminate python altogether. I am not a fan of using layers of different software mostly when I have no control over as is the case with this Anaconda. Thank you!

[drag05]

@dfalbel Re. backward (in)compatibility:

My first question was trying to establish why backward incompatible versions of packages are being released. Your answer seems circular.

Any way, I wonder if I could load the datasets from "Deep Learning with R" using R directly.

Thank you!

[dfalbel]

I am looking for a R versions of keras/ tensorflow to eliminate python altogether.

It's not possible to use TensorFlow or Keras without the Python layer. All the R packages uses reticulate to access python and in turn TensorFlow and Keras (python) packages.

My first question was trying to establish why backward incompatible versions of packages are being released. Your answer seems circular.

The numpy team released an update that broke TensorFlow sometime ago. The workaround before tensorflow released an update was to downgrade numpy.

[drag05]

@dfalbel

I know but I find layered software cumbersome and problem-prone: I'm spending more time troubleshooting and googling than reading the "Deep Learning with R" book: this morning I had a new problem with "not being able to find the _cudart64100.dll" error which, I knew I had solved before.

Then I realized that after installing tensorflow-gpu v. 2.0.0 I needed to re-run keras::install_keras(). Now it seems to be working.

Finally, I have been able to run all datasets including the Boston Housing dataset from "Deep Learning with R"!

I will refrain from updating anything until I finish the book. I also hope that H2O4GPU will find soon support for Windows OS. Thank you!

[drag05]

@dfalbel

Hello Daniel,

After installing (as described above), tensorflow_gpu v.2.0.0 worked for a while then, tensorflow::tf_config() gave this output: 2019-10-11 23:29:03.178012: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found

TensorFlow v2.0.0 (C:\Anaconda3\envs\r-reticulate\lib\site-packages\tensorflow__init__.p) Python v3.6 (C:\Anaconda3\envs\r-reticulate\python.exe)

Please advise. Thank you!

[mjgraham713]

Was this ever resolved? I am running into the same issues trying to load any datasets included in Keras.

[drag05]

@mjgraham713:

With no pip insall, the following steps solved the issue on my machine:

NOTE1: Updating NVIDIA drivers first should be a must!

1. Uninstalled all CUDA Tools and removed their paths in Windows Environment Variables
2. Replaced Anaconda with Miniconda3 (which brings only the base environment)
3. Installed R package Keras

4. From Miniconda prompt ONLY: created the tensorflow environment using the following command:

   conda create --name tensorflow tensorflow-gpu=2.0.0

This command took care of tensorflow/tensorflow-gpu/python version match and also installed cudatools including cudnn within the tensorflow environment, as well as wrote respective paths in Windows Environment Variables.

NOTE2: Updating R package reticulate allowed running deep learning on GPU from R console as well as from R Studio.

NOTE3: Periodically updated all packages in the tensorflow environment

This configuration allowed me to go through CNN and LSTM/GRU without problems.

So, to answer your question, I think the problem was resolved.

Thank you!