Custom negative metric crash

[Laurae2]

Using a custom loss metric crashes R with negative y_true / y_pred.

Commit: fb54c2d Example used: Reuters: https://rstudio.github.io/keras/articles/examples/reuters_mlp.html

See comments and code below:

library(keras)
K <- backend() # manual add-on

max_words <- 1000
batch_size <- 32
epochs <- 5

cat('Loading data...\n')
reuters <- dataset_reuters(num_words = max_words, test_split = 0.2)
x_train <- reuters$train$x
y_train <- reuters$train$y
x_test <- reuters$test$x
y_test <- reuters$test$y

cat(length(x_train), 'train sequences\n')
cat(length(x_test), 'test sequences\n')

num_classes <- max(y_train) + 1
cat(num_classes, '\n')

cat('Vectorizing sequence data...\n')

tokenizer <- text_tokenizer(num_words = max_words)
x_train <- sequences_to_matrix(tokenizer, x_train, mode = 'binary')
x_test <- sequences_to_matrix(tokenizer, x_test, mode = 'binary')

cat('x_train shape:', dim(x_train), '\n')
cat('x_test shape:', dim(x_test), '\n')

cat('Convert class vector to binary class matrix',
    '(for use with categorical_crossentropy)\n')
y_train <- to_categorical(y_train, num_classes)
y_test <- to_categorical(y_test, num_classes)
cat('y_train shape:', dim(y_train), '\n')
cat('y_test shape:', dim(y_test), '\n')

# CUSTOM METRIC CRASH
metric_mean_pred <- function(y_true, y_pred) {
  K$mean(y_true - y_pred) 
}

# the following does not crash
# metric_mean_pred <- function(y_true, y_pred) {
#   K$mean(y_pred) 
# }
# metric_mean_pred <- function(y_true, y_pred) {
#   K$mean(y_true) 
# }
# metric_mean_pred <- function(y_true, y_pred) {
#   K$mean(y_true + y_pred) 
# }

# Even more weird as this one does NOT crash, we have just used addition and changed sign of y_pred
# metric_mean_pred <- function(y_true, y_pred) {
#   K$mean(y_true + -y_pred) 
# }

cat('Building model...\n')
model <- keras_model_sequential()
model %>%
  layer_dense(units = 512, input_shape = c(max_words)) %>% 
  layer_activation(activation = 'relu') %>% 
  layer_dropout(rate = 0.5) %>% 
  layer_dense(units = num_classes) %>% 
  layer_activation(activation = 'softmax')

model %>% compile(
  loss = 'categorical_crossentropy',
  optimizer = 'adam',
  metrics = c('accuracy', 
              'mean_pred' = metric_mean_pred)
)

history <- model %>% fit(
  x_train, y_train,
  batch_size = batch_size,
  epochs = epochs,
  verbose = 1,
  validation_split = 0.1
)

score <- model %>% evaluate(
  x_test, y_test,
  batch_size = batch_size,
  verbose = 1
)

cat('Test score:', score[[1]], '\n')
cat('Test accuracy', score[[2]], '\n')

Another question: should we define axis=-1 for custom metrics like in Python?

Example for mean squared error:

def mean_squared_error(y_true, y_pred):
  return K.mean(K.square(y_pred - y_true), axis=-1)

This one crashes in R:

metric_mse <- function(y_true, y_pred) {
  return K$mean(K$square(y_pred + -y_true), axis = -1)
}

[kevinushey]

FWIW, when I run your example I see this error:

> model %>% compile(
+   loss = 'categorical_crossentropy',
+   optimizer = 'adam',
+   metrics = c('accuracy',
+               'mean_pred' = metric_mean_pred)
+ )
Error in py_call_impl(callable, dots$args, dots$keywords) :
  RuntimeError: Evaluation error: AttributeError: 'module' object has no attribute 'sub'.

Detailed traceback:
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/keras/models.py", line 781, in compile
    **kwargs)
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/keras/engine/training.py", line 898, in compile
    metric_result = masked_metric_fn(y_true, y_pred, mask=masks[i])
  File "/usr/local/opt/python/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/keras/engine/training.py", line 494, in masked
    score_array = fn(y_true, y_pred)
  File "/Users/kevin/Library/R/3.4/library/reticulate/python/rpytools/call.py", line 21, in python_function
    raise RuntimeError(res[kErrorKey])
Calls: %>% ... <Anonymous> -> compile -> <Anonymous> -> py_call_impl -> .Call
Execution halted

Is this what you're referring to, or are you seeing some other kind of crash / error? What's especially odd is my second attempt to compile the model seems to work; it's just the first that emits the error.

[kevinushey]

Can you also provide the output of reticulate::py_config()? In my case:

> reticulate::py_config()
python:         /usr/local/bin/python
libpython:      /usr/local/opt/python/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config/libpython2.7.dylib
pythonhome:     /usr/local/opt/python/Frameworks/Python.framework/Versions/2.7:/usr/local/opt/python/Frameworks/Python.framework/Versions/2.7
version:        2.7.13 (default, Jul 18 2017, 09:17:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
numpy:          /Users/kevin/Library/Python/2.7/lib/python/site-packages/numpy
numpy_version:  1.13.1
keras:          /usr/local/lib/python2.7/site-packages/keras

python versions found: 
 /usr/local/bin/python
 /usr/bin/python
 /usr/local/bin/python3
 /Users/kevin/.virtualenvs/r-tensorflow/bin/python

[Laurae2]

Is this what you're referring to, or are you seeing some other kind of crash / error?

I'm referring to this:

Can you also provide the output of reticulate::py_config()? In my case:

Here below:

> reticulate::py_config()
python:         C:\PROGRA~1\ANACON~1\python.exe
libpython:      C:/PROGRA~1/ANACON~1/python35.dll
pythonhome:     C:\PROGRA~1\ANACON~1
version:        3.5.3 |Anaconda 4.4.0 (64-bit)| (default, May 15 2017, 10:43:23) [MSC v.1900 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:\PROGRA~1\ANACON~1\lib\site-packages\numpy
numpy_version:  1.12.1

python versions found: 
 C:\PROGRA~1\ANACON~1\python.exe
 C:\Program Files\Anaconda3\envs\r-tensorflow\python.exe

R is run with administrator rights.

[kevinushey]

Got it; thanks! I see a plain R error on my macOS machine; maybe that same error is causing a hard crash on Windows?

It seems like this could be some kind of module lazy-loading error (or similar). On my machine, the compile() call actually runs successfully the second time I run it, which is surprising. I'll dig in a little more.

[kevinushey]

Can you try the following and let me know if it resolves the issue? If so, that should help us pinpoint what needs to change in the keras / tensorflow packages.

# CUSTOM METRIC CRASH
metric_mean_pred <- function(y_true, y_pred) {
  force(tf$subtract)  # force initialization of `tf$subtract` method
  K$mean(y_true - y_pred) 
}

[Laurae2]

Can you try the following and let me know if it resolves the issue? If so, that should help us pinpoint what needs to change in the keras / tensorflow packages.

Still crashes:

[kevinushey]

Does it make a difference if you explicitly attach the tensorflow package with library(tensorflow) before running the Keras code here?

[kevinushey]

My hypothesis thus far: there's some lazy-loading of the TensorFlow Python module happening here; for some reason, the TensorFlow R-level module hasn't been loaded yet at the time when the custom metric function is invoked, and that causes an R error to be emitted when the custom metric is called.

Unfortunately, this then causes an exception / long jump across module boundaries which triggers an R session crash. (Note to self: we likely want to wrap the invocation of a custom module function in an R tryCatch handler, to ensure that R errors don't try to leak across module boundaries -- at least on Windows)

[kevinushey]

This version of your script runs fine on my Windows machine:

library(tensorflow)
library(keras)
tf$VERSION
K <- backend() # manual add-on

max_words <- 1000
batch_size <- 32
epochs <- 5

cat('Loading data...\n')
reuters <- dataset_reuters(num_words = max_words, test_split = 0.2)
x_train <- reuters$train$x
y_train <- reuters$train$y
x_test <- reuters$test$x
y_test <- reuters$test$y

cat(length(x_train), 'train sequences\n')
cat(length(x_test), 'test sequences\n')

num_classes <- max(y_train) + 1
cat(num_classes, '\n')

cat('Vectorizing sequence data...\n')

tokenizer <- text_tokenizer(num_words = max_words)
x_train <- sequences_to_matrix(tokenizer, x_train, mode = 'binary')
x_test <- sequences_to_matrix(tokenizer, x_test, mode = 'binary')

cat('x_train shape:', dim(x_train), '\n')
cat('x_test shape:', dim(x_test), '\n')

cat('Convert class vector to binary class matrix',
    '(for use with categorical_crossentropy)\n')
y_train <- to_categorical(y_train, num_classes)
y_test <- to_categorical(y_test, num_classes)
cat('y_train shape:', dim(y_train), '\n')
cat('y_test shape:', dim(y_test), '\n')

# CUSTOM METRIC CRASH
metric_mean_pred <- function(y_true, y_pred) {
  K$mean(y_true - y_pred) 
}

# the following does not crash
# metric_mean_pred <- function(y_true, y_pred) {
#   K$mean(y_pred) 
# }
# metric_mean_pred <- function(y_true, y_pred) {
#   K$mean(y_true) 
# }
# metric_mean_pred <- function(y_true, y_pred) {
#   K$mean(y_true + y_pred) 
# }

# Even more weird as this one does NOT crash, we have just used addition and changed sign of y_pred
# metric_mean_pred <- function(y_true, y_pred) {
#   K$mean(y_true + -y_pred) 
# }

cat('Building model...\n')
model <- keras_model_sequential()
model %>%
  layer_dense(units = 512, input_shape = c(max_words)) %>% 
  layer_activation(activation = 'relu') %>% 
  layer_dropout(rate = 0.5) %>% 
  layer_dense(units = num_classes) %>% 
  layer_activation(activation = 'softmax')

model %>% compile(
  loss = 'categorical_crossentropy',
  optimizer = 'adam',
  metrics = c('accuracy', 
              'mean_pred' = metric_mean_pred)
)

history <- model %>% fit(
  x_train, y_train,
  batch_size = batch_size,
  epochs = epochs,
  verbose = 1,
  validation_split = 0.1
)

score <- model %>% evaluate(
  x_test, y_test,
  batch_size = batch_size,
  verbose = 1
)

cat('Test score:', score[[1]], '\n')
cat('Test accuracy', score[[2]], '\n')

Note the explicit loading of TensorFlow + querying of the TensorFlow VERSION field (which forces a load of the tf module, which gets implicitly used by your custom metric).

For the record:

> reticulate:::py_config()
python:         C:\Users\kevin\Anaconda3\envs\r-tensorflow\python.exe
libpython:      C:/Users/kevin/Anaconda3/envs/r-tensorflow/python36.dll
pythonhome:     C:\Users\kevin\ANACON~1\envs\R-TENS~1
version:        3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 12:30:02) [MSC v.1900 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:\Users\kevin\ANACON~1\envs\R-TENS~1\lib\site-packages\numpy
numpy_version:  1.13.1
keras:          C:\Users\kevin\ANACON~1\envs\R-TENS~1\lib\site-packages\keras

python versions found: 
 C:\Users\kevin\ANACON~1\python.exe
 C:\Users\kevin\Anaconda3\envs\r-tensorflow\python.exe

[Laurae2]

When adding library(tensorflow) on top of it, it still crashes.

However, when using both library(tensorflow) and your fix force(tf$subtract), it compiles and runs the model successfully.

[kevinushey]

Awesome! That's progress, at least.

What version of the RStudio IDE are you using? I don't see an R session crash on my Windows machine when running with R 3.4.1 and RStudio v1.1.338 (the current preview release); it's possible that the hard crash (rather than R error) is better handled in the latest version of RStudio.

[jjallaire]

This should be resolved with https://github.com/rstudio/reticulate/pull/89

You can re-install Keras from GitHub to get the fix. Note that when installing new versions of reticulate you need to be extremely careful to do this in a fresh R session (with no other R sessions that use reticulate loaded). This is because R will more or less silently appear to complete installations of packages where the package DLL is in use and can't be overwritten. So exit all R sessions then do this in a fresh session:

devtools::install_github("rstudio/keras")

[Laurae2]

@kevinushey Using RStudio 1.0.143, but it also crashes with 1.1.338 preview release.

@jjallaire I reinstalled Keras from devtools from a fully fresh session (it downloads also the latest reticulate from GitHub), and now I am getting this:

> library(keras)
> K <- backend()
Error in py_has_attr_impl(x, name) : 
  object '_reticulate_py_has_attr_impl' not found
In addition: Warning message:
In stop(e, call. = FALSE) : additional arguments ignored in stop()

It causes a crash when trying to get data:

reuters <- dataset_reuters(num_words = max_words, test_split = 0.2)

[jjallaire]

The error message you are reporting indicates that the old version of reticulate is still in use on your system. You should shut down all R sessions and then do the install in a fresh session. Perhaps consider rebooting before the install.

On Tue, Aug 22, 2017 at 2:49 AM, Laurae notifications@github.com wrote:

@kevinushey https://github.com/kevinushey Using RStudio 1.0.143, but it also crashes with 1.1.338 preview release.

@jjallaire https://github.com/jjallaire I reinstalled Keras from devtools from a fully fresh session (it downloads also the latest reticulate from GitHub), and now I am getting this:

library(keras)> K <- backend()Error in py_has_attr_impl(x, name) : object '_reticulate_py_has_attr_impl' not foundIn addition: Warning message:In stop(e, call. = FALSE) : additional arguments ignored in stop()

It causes a crash when trying to get data:

reuters <- dataset_reuters(num_words = max_words, test_split = 0.2)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rstudio/keras/issues/108#issuecomment-323961532, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGXx9ukttgLzDAnsDWg5auPGGxAiZzjks5sapYCgaJpZM4O9e65 .