custom metric and saved models

[dambuck]

Hi,

so working with version 0.4.4 now.

There are a few things which are still unclear to me when using custom metrics and trying to load the saved_models.

def GRU_test(x_train,y_train,x_val,y_val, params):
    neuron=73
    latent_rep_size=73
    def vae_loss(true, pred):
        true = K.flatten(true)
        pred = K.flatten(pred)
        poisson = 1*objectives.poisson(true, pred)
    return poisson 

def R2_metric(y_true, y_pred):
    SS_res =  K.sum(K.square(y_true-y_pred)) 
    SS_tot = K.sum(K.square(y_true - K.mean(y_true))) 
    return (1 - SS_res/(SS_tot + K.epsilon()))

inp = Input(shape=(time_steps, neuron)) 
encode = Bidirectional(GRU(neuron, return_sequences=True, dropout=0.0, name='lstm_1'), merge_mode='concat')(inp)
encode = Bidirectional(GRU(neuron, activation='linear', dropout=0.0, name='lstm_2'), merge_mode='concat')(encode)

z_mean = Dense(neuron, name='z_mean', activation='linear')(encode)
z_log_var = Dense(neuron, name='z_log_var', activation='linear')(encode)
encode=Lambda(sampling, output_shape=(neuron,), name='lambda')([z_mean, z_log_var])

repeated_context = RepeatVector(time_steps)(encode)

decode = GRU(neuron, return_sequences=True,dropout=0.05, name='dec_lstm_12',activation='linear')(repeated_context)

decode=TimeDistributed(Dense(params['size'], activation='linear'), name='get_factors')(decode)
decode=TimeDistributed(Dense(neuron, activation='linear'), name= 'pre_rates')(decode)
decode=TimeDistributed(Dense(neuron, activation= K.exp ),name= 'exp_layer')(decode)

nan=TerminateOnNaN()
callbacklist=[nan]

adam = optimizers.adam(clipvalue=200)
Vae=Model(inputs=inp, outputs=decode)
Vae.compile(optimizer='adam',loss= vae_loss, metrics=[R2_metric])
out=Vae.fit(x_train, y_train,validation_data=[x_val,y_val],epochs=400, callbacks=callbacklist)

return out, Vae

Gru = ta.Scan(ofc_binned, ofc_binned, params=latent_size,dataset_name='GRU_log', model=GRU_test,experiment_no='1')

One thing is the load_model function which is listed as part of the Predict() command in the talos manual. I just couldnt manage to pick a specific model to load and to test. It gives me the error:

<ipython-input-36-26b8b8f6ad51> in <module>()
  1 p = ta.Predict(Gru)
  2 # p.predict(ofc_binned,0)
----> 3 p.load_model(0)

AttributeError: 'Predict' object has no attribute 'load_model'

When just using the p.predict(data,1) line, I get an error for unknown function exp. How do I add a custom object dict at this point?

When trying something like saving the model first and loading it:

settings_dict={'exp': K.exp, 'latent_rep_size':73,'epsilon_std':1 }
one=model_from_json(saved_models[0],custom_objects=settings_dict)
one.save(os.getcwd()+ '/trained/GRU_test/model.hdf5' )
one = load_model(os.getcwd()+'/trained/GRU_test/model.hdf5' ,custom_objects=settings_dict)

When predicting with that model I clearly get the results of an untrained model. Does talos actually save all the models that are tested? If so could you show me a quick way how to load them?

Thanks a lot

[mikkokotila]

Did you check the user guide in the corresponding part? I think the example there will help solve your problem. Also check out the corresponding docstrings for the functions for more.

For the case where for some reason you want to get access to all the models, that is also available through the Scan() object (but I'm guessing that's not what you're looking for now).

[dambuck]

Thanks for the fast answer.

I already checked it, and I found it weird that predict is listed as part of the Predict command but the manual shows it's use as part of the Reporting command. Might be an error?

I might be just too stupid but I cant manage to get the predict command to work, because of the exp activation function... How do I declare this custom object here?

p = ta.Predict(Gru)
p.predict(ofc_binned,0)

ValueError                                Traceback (most recent call last)
<ipython-input-30-fadcb4245a59> in <module>()
      1 custom_objects=settings_dict
      2 p = ta.Predict(Gru)
----> 3 p.predict(ofc_binned,0)
      4 # p.load_model()

~/.local/lib/python3.5/site-packages/talos/commands/predict.py in predict(self, x, model_id, metric, asc)
     27             model_id = best_model(self.scan_object, metric, asc)
     28 
---> 29         model = activate_model(self.scan_object, model_id)
     30 
     31         return model.predict(x)

~/.local/lib/python3.5/site-packages/talos/utils/best_model.py in activate_model(self, model_id)
     18     '''Loads the model from the json that is stored in the Scan object'''
     19 
---> 20     model = model_from_json(self.saved_models[model_id])
     21     model.set_weights(self.saved_weights[model_id])
     22 

~/.local/lib/python3.5/site-packages/keras/engine/saving.py in model_from_json(json_string, custom_objects)
    366     config = json.loads(json_string)
    367     from ..layers import deserialize
--> 368     return deserialize(config, custom_objects=custom_objects)
    369 
    370 

~/.local/lib/python3.5/site-packages/keras/layers/__init__.py in deserialize(config, custom_objects)
     53                                     module_objects=globs,
     54                                     custom_objects=custom_objects,
---> 55                                     printable_module_name='layer')

~/.local/lib/python3.5/site-packages/keras/utils/generic_utils.py in deserialize_keras_object(identifier,    module_objects, custom_objects, printable_module_name)
    143                     config['config'],
    144                     custom_objects=dict(list(_GLOBAL_CUSTOM_OBJECTS.items()) +
--> 145                                         list(custom_objects.items())))
    146             with CustomObjectScope(custom_objects):
    147                 return cls.from_config(config['config'])

~/.local/lib/python3.5/site-packages/keras/engine/network.py in from_config(cls, config, custom_objects)
   1015         # First, we create all layers and enqueue nodes to be processed
   1016         for layer_data in config['layers']:
-> 1017             process_layer(layer_data)
   1018         # Then we process nodes in order of layer depth.
   1019         # Nodes that cannot yet be processed (if the inbound node

~/.local/lib/python3.5/site-packages/keras/engine/network.py in process_layer(layer_data)
   1001 
   1002             layer = deserialize_layer(layer_data,
-> 1003                                       custom_objects=custom_objects)
   1004             created_layers[layer_name] = layer
   1005 

~/.local/lib/python3.5/site-packages/keras/layers/__init__.py in deserialize(config, custom_objects)
     53                                     module_objects=globs,
     54                                     custom_objects=custom_objects,
---> 55                                     printable_module_name='layer')

~/.local/lib/python3.5/site-packages/keras/utils/generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
    143                     config['config'],
    144                     custom_objects=dict(list(_GLOBAL_CUSTOM_OBJECTS.items()) +
--> 145                                         list(custom_objects.items())))
    146             with CustomObjectScope(custom_objects):
    147                 return cls.from_config(config['config'])

~/.local/lib/python3.5/site-packages/keras/layers/wrappers.py in from_config(cls, config, custom_objects)
    108         from . import deserialize as deserialize_layer
    109         layer = deserialize_layer(config.pop('layer'),
--> 110                                   custom_objects=custom_objects)
    111         return cls(layer, **config)
    112 

~/.local/lib/python3.5/site-packages/keras/layers/__init__.py in deserialize(config, custom_objects)
     53                                     module_objects=globs,
     54                                     custom_objects=custom_objects,
---> 55                                     printable_module_name='layer')

~/.local/lib/python3.5/site-packages/keras/utils/generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
    145                                         list(custom_objects.items())))
    146             with CustomObjectScope(custom_objects):
--> 147                 return cls.from_config(config['config'])
    148         else:
    149             # Then `cls` may be a function returning a class.

~/.local/lib/python3.5/site-packages/keras/engine/base_layer.py in from_config(cls, config)
   1107             A layer instance.
   1108         """
-> 1109         return cls(**config)
   1110 
   1111     def count_params(self):

~/.local/lib/python3.5/site-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
     89                 warnings.warn('Update your `' + object_name +
    90                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 91             return func(*args, **kwargs)
     92         wrapper._original_function = func
     93         return wrapper

~/.local/lib/python3.5/site-packages/keras/layers/core.py in __init__(self, units, activation, use_bias,         kernel_initializer, bias_initializer, kernel_regularizer, bias_regularizer, activity_regularizer, kernel_constraint, bias_constraint, **kwargs)
   839         super(Dense, self).__init__(**kwargs)
    840         self.units = units
--> 841         self.activation = activations.get(activation)
    842         self.use_bias = use_bias
    843         self.kernel_initializer = initializers.get(kernel_initializer)

~/.local/lib/python3.5/site-packages/keras/activations.py in get(identifier)
    187     if isinstance(identifier, six.string_types):
    188         identifier = str(identifier)
--> 189         return deserialize(identifier)
   190     elif callable(identifier):
    191         if isinstance(identifier, Layer):

~/.local/lib/python3.5/site-packages/keras/activations.py in deserialize(name, custom_objects)
   168         module_objects=globals(),
    169         custom_objects=custom_objects,
--> 170         printable_module_name='activation function')
   171 
    172 

~/.local/lib/python3.5/site-packages/keras/utils/generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name)
    163             if fn is None:
    164                 raise ValueError('Unknown ' + printable_module_name +
--> 165                                  ':' + function_name)
    166         return fn
    167     else:

ValueError: Unknown activation function:exp

[mikkokotila]

How about the example [here](https://nbviewer.jupyter.org/github/autonomio/talos/blob/master/examples/Hyperparameter%20Optimization%20with%20Keras%20for%20the%20Iris%20Prediction.ipynb#7.-Evaluating-Models-with-Evaluate()-), would that help you to get your head around it?

There might be some inconsistency with the information, so I will check into that. Sorry for inconvenience.

Generally speaking the idea is that in order to actually use the models, there are several different ways in which we can do that within Talos, but the idea is to avoid entirely handling the save / load outside of Talos. The most seamless of these is the Deploy() and then working out of the deploy package (there is a readme included in the deploy package that explains how).

If the above does not help, I can create an example notebook which is just focused on making predictions with models.