Can't load a model after having saved

[nandofernandesneto]

Hi!

After saving a MDN model estimated using TF Probability this way:

model.MDN %>% save_model_hdf5("/home/fernando/dunamis/deep_model.h5")

When I try to load it on a new R session, I've got the following error:

model.MDN <- load_model_hdf5(paste0(path, "deep_model.h5"))
Error in py_call_impl(callable, dots$args, dots$keywords) : 
  ValueError: Unknown layer: DistributionLambda

Any ideas?

Thanks in advance.

Best Regards,

Fernando Fernandes

[skeydan]

Hi,

for layer_distribution_lambda, like for layer_lambda, you have to recreate the graph programmatically. You can save the model weights using

save_model_weights_tf

and restore them using

load_model_weights_tf.

See also this vignette as to model saving https://keras.rstudio.com/articles/saving_serializing.html

[nandofernandesneto]

Hello!

Thank you very much for your reply.

When I try to use save_model_weights_tf I get:

Error in save_model_weights_tf(model.MDN, "/home/fernando/dunamis/deep_model/") : Save weights to the SavedModel format requires TensorFlow version >= 2.0

I thought this was supported on tensorflow 1.14

[nandofernandesneto]

Are these packages (tfestimators and tfprobability) compatible with Tensorflow 2.0?

[skeydan]

@nandofernandesneto yes you will have to use TensorFlow 2.0 beta.

If you install a version that matches your OS from https://pypi.org/project/tf-nightly-2.0-preview/#files, doing e.g.

tensorflow::install_tensorflow(version = "https://files.pythonhosted.org/packages/6e/d4/cbbb28c1a45a37eccf0a3b01a10673daad237a4f04d66730eb63e8e21aa2/tf_nightly_2.0_preview-2.0.0.dev20190729-cp37-cp37m-manylinux1_x86_64.whl", method = "conda", envname = "myenv")

the above commands will work, but there is something to know in addition, which is that after restoring the weights and before calling the model for prediction, the input data need to be converted to tf$float32, like so

model %>% load_model_weights_tf("/tmp/k")
# don't do this
# model(x)
# instead do this 
model(tf$cast(x, tf$float32))

@dfalbel adding you here, what we talked about yesterday (dynamic types due to eager)... So this is pretty, ehm, interesting ... and applies not to TFP only but to pure TF as well.

To reproduce, here it is without TFP:

# use some tf 2 preview
reticulate::use_condaenv("b")

library(tensorflow)
library(keras)

x <- matrix(1.1:10.1, ncol = 1)
y <- x

model2 <- keras_model_sequential() %>%
  layer_dense(units = 1, activation = "linear")
model2 %>% compile(optimizer = optimizer_adam(lr = 0.1), loss = "mse")
model2 %>% fit(x, y, epochs = 100)
model2 %>% save_model_weights_tf("/tmp/k2")

# now restart R and execute all of the above besides the last 2 steps

model2 %>% load_model_weights_tf("/tmp/k2")

# now in version 1, first call the model without the cast, then with the cast
# both calls with throw the type error
model2(x)
model2(tf$cast(x, tf$float32))

# now in version 2 do it other way round
# both calls will work
model2(tf$cast(x, tf$float32))
model2(x)

This is something we should think about I guess (possibly wrap the call)? I haven't tested though what happens with predict instead of _call_ing the model.

[nandofernandesneto]

Not willing to abuse of your knowledge and time, my models are making use of tfprobability and tfestimators. Are the current R implementations of them compatible with TF 2.0?

[nandofernandesneto]

UPDATE: After upgrading my tensorflow to 1.16, this seems to work:

save_model_weights_hdf5(model.MDN, "/home/fernando/dunamis/deep_model.h5")

Afterwards, we have recreated the graph, in another session and loaded the weights again using:

model.MDN %>% load_model_weights_hdf5("/home/fernando/dunamis/deep_model.h5")

It might be useful for someone else using tfprobability elsewhere.

We were able to use predict and so on... So, it seems to be working.

[skeydan]

Thanks, I'll close then :-)