Loading a model trained in python

[beamandrew]

I have a pretty simple RNN that was trained in Python on a big GPU server that I'd like to deploy in a shiny app.

The model looks like this:

input = Input(shape=input_shape, dtype='int32', name='main_input')
x = Embedding(output_dim=embedding_matrix.shape[1], input_dim=vocab_size, input_length=input_len, weights=[embedding_matrix])(input)
x = Dropout(embedding_dropout)(x)
x = GRU(recurrent_size, recurrent_dropout=recurrent_dropout, dropout=gru_dropout,kernel_regularizer=regularizers.l2(GRU_l2_penalty))(x)
x = Dense(dense_size,activation='relu',kernel_regularizer=regularizers.l2(dense_l2_penalty))(x)
x = Dropout(dense_dropout)(x)
predictions = Dense(self.num_classes, activation='softmax')(x)

model = Model(input, predictions)
opt = Adam(lr=learning_rate)
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=["accuracy", top_k_categorical_accuracy])

After training, I save the model + weights to HDF5 using:

model.save('my_model.h5')

When I try to load this model in R, using load_model_hdf5, I get the following error:

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor("embedding_4_1/random_uniform:0", shape=(102411, 500), dtype=float32)'

Detailed traceback: 
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/models.py", line 239, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/models.py", line 313, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/layers/__init__.py", line 54, in deserialize
    printable_module_name='layer')
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 139, in deserialize_keras_object
    list(custom_objects.items())))
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras

Is it currently possible to load models that were trained with the python version of Keras? If so, how should I transfer a model from python to R?

[jjallaire]

Yes, you should definitely be able to load a model trained in Python within R!

If you could provide an example that is runnable for me locally (it can just use whatever fake data you want it to) then I'm sure I can get to the bottom of what is going wrong here.

[jjallaire]

What if you try this from the same R session where you are seeing this error:

reticulate::py_run_string("import keras; keras.models.load_model('my_model.h5')")

If this fails with the same error then perhaps the version of Keras used for training doesn't match the version of Keras used with R and that is the source of the problem?

[beamandrew]

Thank you for the very fast response!

Here is a minimal example that recreates the error without training the model:

from keras.layers import Dense, Embedding, GRU, Bidirectional, Dropout
from keras.models import Model, Input
from keras import regularizers
from keras.optimizers import Adam
from keras.metrics import top_k_categorical_accuracy

embedding_dropout = 0.5
recurrent_dropout = 0.5
gru_dropout = 0.1
dense_dropout = 0.5
GRU_l2_penalty = 1.0e-5
dense_l2_penalty = 1.0e-5
learning_rate = 0.0007
recurrent_size = 64
dense_size = 512

input = Input(shape=(100,), dtype='int32', name='main_input')
x = Embedding(output_dim=500, input_dim=50000, input_length=100)(input)
x = Dropout(embedding_dropout)(x)
x = GRU(recurrent_size, recurrent_dropout=recurrent_dropout, dropout=gru_dropout,kernel_regularizer=regularizers.l2(GRU_l2_penalty))(x)
x = Dense(dense_size,activation='relu',kernel_regularizer=regularizers.l2(dense_l2_penalty))(x)
x = Dropout(dense_dropout)(x)
predictions = Dense(517, activation='softmax')(x)

model = Model(input, predictions)
opt = Adam(lr=learning_rate)
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=["accuracy", top_k_categorical_accuracy])

model.save("my_model.h5")

Then when I try to load in R:

library(keras)
load_model_hdf5("my_model.h5")

I get:

> load_model_hdf5("~/Downloads/rnn_model.h5")
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor("embedding_4_5/random_uniform:0", shape=(50000, 500), dtype=float32)'

Detailed traceback: 
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/models.py", line 239, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/models.py", line 313, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/layers/__init__.py", line 54, in deserialize
    printable_module_name='layer')
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 139, in deserialize_keras_object
    list(custom_objects.items())))
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/

What if you try this from the same R session where you are seeing this error:

I still see the same error. The python version of Keras is 2.0.2 and it is the GPU version. The R version I'm using keras_2.0.7.9008 and is the CPU version obtained from devtools::install_github("rstudio/keras") followed by install_keras()

[jjallaire]

Okay, that means that the issue is Keras version skew (i.e. it doesn't have anything to do with the fact that you are loading the model into R). The version skew could be due to using the GPU vs. using the CPU, or it could be usage of Keras 2.0.2 when saving the model and Keras 2.0.8 when loading the model. I'll do some additional investigation here to try to narrow it down, let me know what you discover on your end.

[jjallaire]

Just to fully establish that this isn't an issue of the R bindings, you could try this from a bash prompt:

source ~/.virtualenvs/r-tensorflow/bin/activate
python -c'import keras; keras.models.load_model("my_model.h5")'

[beamandrew]

It appears to be the same error:

(r-tensorflow) andy$ python -c'import keras; keras.models.load_model("my_model.h5")'
Using TensorFlow backend.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/models.py", line 239, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/models.py", line 313, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/layers/__init__.py", line 54, in deserialize
    printable_module_name='layer')
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 139, in deserialize_keras_object
    list(custom_objects.items())))
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/engine/topology.py", line 2497, in from_config
    process_node(layer, node_data)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/engine/topology.py", line 2454, in process_node
    layer(input_tensors[0], **kwargs)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/engine/topology.py", line 575, in __call__
    self.build(input_shapes[0])
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/layers/embeddings.py", line 101, in build
    dtype=self.dtype)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 87, in wrapper
    return func(*args, **kwargs)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/engine/topology.py", line 399, in add_weight
    constraint=constraint)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 316, in variable
    v = tf.Variable(value, dtype=_convert_string_dtype(dtype), name=name)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 199, in __init__
    expected_shape=expected_shape)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 289, in _init_from_args
    initial_value, name="initial_value", dtype=dtype)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 611, in convert_to_tensor
    as_ref=False)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 676, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 549, in _TensorTensorConversionFunction
    % (dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype int32 for Tensor with dtype float32: 'Tensor("embedding_4/random_uniform:0", shape=(50000, 500), dtype=float32)'

[jjallaire]

Okay, so that confirms this is Keras version skew issue.

[jjallaire]

I don't see the error when saving w/ the GPU version and then loading with the CPU version, so it must be Keras v2.02 vs. Keras v2.0.8

[beamandrew]

If I update my python version Keras to 2.0.8 do you think that will help? I don't think there have been any API breaking changes, but I'm afraid of chasing my tail and breaking other things in the pipeline.

[jjallaire]

You can arrange to have R use an older version of Keras if you know which Python environment on your machine has it. For example, let's say you have a virtualenv named "keras-2.0.2" at ~/.virtualenvs/keras-2.0.2". The following would cause R to use this version of Keras:

library(keras)
use_virtualenv("keras-2.0.2")

Docs on this are here; https://rstudio.github.io/reticulate/articles/versions.html

[jjallaire]

I've independently confirmed that saving the model in Keras 2.0.2 and then loading it in Keras 2.0.8 is the source of the error. On the same machine saving and loading in Keras 2.0.8 works correctly, so I'm pretty confident that updating your training environment to Keras 2.0.8 will do the job.

Of course you can also arrange to use Keras 2.0.2 with R as follows:

cd ~/.virtualenvs
virtualenv keras-2.0.2
source keras-2.0.2/bin/activate
pip install keras==2.0.2
pip install tensorflow

Then from R

library(keras)
use_virtualenv("keras-2.0.2")

[beamandrew]

Thanks! I decided to bite the bullet and upgrade. In the process of upgrading now and will report back once complete.

[beamandrew]

After upgrading it appears to load correctly! Thanks for the very rapid response and thanks for this amazing package!

[jjallaire]

Terrific, glad that all is well!

Is the Shiny app you are building something you could share? (we are looking for case studies to highlight building Shiny front-ends to Keras models). @jcheng5

[beamandrew]

I definitely plan to share, but it's not ready yet. I actually intend to put it up on on shinyapps.io when it is all complete. I'm hoping to put it up sometime next week.

Andy

On Wed, Sep 27, 2017 at 1:51 PM JJ Allaire notifications@github.com wrote:

Terrific, glad that all is well!

Is the Shiny app you are building something you could share? (we are looking for case studies to highlight building Shiny front-ends to Keras models). @jcheng5 https://github.com/jcheng5

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/rstudio/keras/issues/141#issuecomment-332603123, or mute the thread https://github.com/notifications/unsubscribe-auth/ADlbameskybMxbxjYO6LS3yqq3AzzHDDks5smos2gaJpZM4PmDUM .

-- Andrew L. Beam, PhD Instructor in Biomedical Informatics Department of Biomedical Informatics Harvard Medical School

[jjallaire]

Okay, great. Ping me when you get it published! BTW shinyapps.io images have built-in support for TF and Keras so the deployment should go smoothly. Let us know if you run into any problems.

[beamandrew]

Awesome, I'll keep you posted! The RStudio team continues to impress.

On Wed, Sep 27, 2017 at 1:57 PM JJ Allaire notifications@github.com wrote:

Okay, great. Ping me when you get it published! BTW shinyapps.io images have built-in support for TF and Keras so the deployment should go smoothly. Let us know if you run into any problems.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/rstudio/keras/issues/141#issuecomment-332604760, or mute the thread https://github.com/notifications/unsubscribe-auth/ADlbat5MylAsUYG6irNg24cZJWGngX6Hks5smoyFgaJpZM4PmDUM .

-- Andrew L. Beam, PhD Instructor in Biomedical Informatics Department of Biomedical Informatics Harvard Medical School

[beamandrew]

I have one more problem trying to load the model. The actual model I'd like to use has a Lambda layer, that implements "Monte Carlo" dropout where the dropout mask is resampled and applied even during test time.

It looks like this:

from keras.layers import Dense, Embedding, GRU, Bidirectional, Dropout
from keras.models import Model, Input
from keras import regularizers
from keras.optimizers import Adam
from keras.metrics import top_k_categorical_accuracy

from keras.layers.core import Lambda
from keras import backend as K

def MC_Dropout(self,p=0.5):
    return Lambda(lambda x: K.dropout(x, level=p))

embedding_dropout = 0.5
recurrent_dropout = 0.5
gru_dropout = 0.1
dense_dropout = 0.5
GRU_l2_penalty = 1.0e-5
dense_l2_penalty = 1.0e-5
learning_rate = 0.0007
recurrent_size = 64
dense_size = 512

input = Input(shape=(100,), dtype='int32', name='main_input')
x = Embedding(output_dim=500, input_dim=50000, input_length=100)(input)
x = MC_Dropout(embedding_dropout)(x)
x = GRU(recurrent_size, recurrent_dropout=recurrent_dropout, dropout=gru_dropout,kernel_regularizer=regularizers.l2(GRU_l2_penalty))(x)
x = Dense(dense_size,activation='relu',kernel_regularizer=regularizers.l2(dense_l2_penalty))(x)
x = MC_Dropout(dense_dropout)(x)
predictions = Dense(517, activation='softmax')(x)

model = Model(input, predictions)
opt = Adam(lr=learning_rate)
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=["accuracy", top_k_categorical_accuracy])

model.save(“mc_dropout_model.h5")

but when I try to load in R

model = load_model_hdf5('mc_dropout_model.h5')

I get the following error

Using TensorFlow backend.
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  TypeError: arg 5 (closure) must be None or tuple

Detailed traceback: 
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/models.py", line 239, in load_model
    model = model_from_config(model_config, custom_objects=custom_objects)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/models.py", line 313, in model_from_config
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/layers/__init__.py", line 54, in deserialize
    printable_module_name='layer')
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 139, in deserialize_keras_object
    list(custom_objects.items())))
  File "/Users/andy/.virtualenvs/r-tensorflow/lib/python2.7/site-packages/keras/engine/topology.py", line 2487, in from_config
    process_layer(layer_data)
  File "/Users/andy/.virtualenvs/r-

What's the correct way to load a model with a custom Lambda layer?

[jjallaire]

Note that the same error occurs if you try to load the model in Python. Try appending the following to the Python script and you get the same error:

from keras.models import load_model
load_model("mc_dropout_model.h5")

My guess is that if you get this working in Python then it will also just work on the R side.

[beamandrew]

Good point! Closing for now assuming that it's a python/keras issue.

[Tixierae]

@jjallaire For what it's worth, here is a Shiny app that exposes a Keras model for text: https://safetyapp.shinyapps.io/DNLPvis/ (work in progress). Right now, the app just calls the model that is hosted on Heroku via a flask app. But since I'm having problems recently with flask/Heroku (TensorFlow memory leaks), I'm considering deploying the model directly inside Shiny (hopefully I won't have the same issue).

[KafeelBasha]

This issue was helpful to me.

I am at a similar situation now, I was able to save the keras model from python and load it into R, basically a text classifier using tfidf/mlp. Now after loading the model into R, how can I transform the new data or text so that I can predict using the model which I just loaded into R.

Would like to convert "Raw text to labels" in R now.

vec=tfidf.transform(data[txt])
l=model.predict_classes(vec)
data['pred_labels']=enc.inverse_transform(list(l))

Any suggestion will be helpful.