Error using Keras optimizers with MXNet backend

[alexrisman]

Hi I've been trying to run Keras with the MXNet backend and it seems to not like Keras optimizers. When I do the following, I get the following error when trying to train: from keras.optimizers import Nadam opt = Nadam(lr=0.0002) gpu_list = ["gpu(0)"] model.compile(optimizer=opt, loss='binary_crossentropy', context=gpu_list) model.fit(X_train, y_train, shuffle=True, batch_size=batch_size, nb_epoch=1)

Traceback (most recent call last): File "train.py", line 157, in train() File "train.py", line 130, in train hist = model.fit(X_train, y_train, shuffle=True, batch_size=batch_size, nb_epoch=1) File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1198, in fit initial_epoch=initial_epoch) File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 893, in _fit_loop outs = f(ins_batch) File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1954, in trainfunction data, label, , data_shapes, label_shapes = self._adjust_module(inputs, 'train') File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1912, in _adjust_module self._mod.init_optimizer(kvstore=self._kvstore, optimizer=self.optimizer) File "/usr/local/lib/python2.7/dist-packages/mxnet/module/bucketing_module.py", line 385, in init_optimizer force_init=force_init) File "/usr/local/lib/python2.7/dist-packages/mxnet/module/module.py", line 509, in init_optimizer assert isinstance(optimizer, opt.Optimizer) AssertionError

The training code runs when I use an MXNet optimizer, but then I can't save models: from mxnet.optimizer import Nadam opt = Nadam(learning_rate=0.0002) gpu_list = ["gpu(0)"] model.compile(optimizer=opt, loss='binary_crossentropy', context=gpu_list) model.fit(X_train, y_train, shuffle=True, batch_size=batch_size, nb_epoch=1) model.save(local_model_file_name, overwrite=True)

Traceback (most recent call last): File "train.py", line 157, in train() File "train.py", line 136, in train model.save(local_model_file_name, overwrite=True) File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2642, in save save_model(self, filepath, overwrite) File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 74, in save_model 'config': model.optimizer.get_config() AttributeError: 'Nadam' object has no attribute 'get_config'

Any thoughts? Thanks.

[laucheukhim]

The Nadam optimizer is not implemented. Here's a patch that I use to get it running. I cannot guarantee the implementation correctness though.

import keras.backend as K
import keras.optimizers
from keras.optimizers import MXOptimizer

class Nadam(MXOptimizer, K.mx.optimizer.Nadam):
    def __init__(self, lr=0.002, beta_1=0.9, beta_2=0.999,
                 epsilon=1e-8, schedule_decay=0.004, clipnorm=None, **kwargs):
        K.mx.optimizer.Nadam.__init__(self, learning_rate=lr, beta1=beta_1, beta2=beta_2,
                                      epsilon=epsilon, schedule_decay=schedule_decay, clip_gradient=clipnorm, **kwargs)
        MXOptimizer.__init__(self, lr, 0.)

    def get_config(self):
        config = {'lr': float(K.get_value(self.lr)),
                  'beta_1': float(K.get_value(self.beta1)),
                  'beta_2': float(K.get_value(self.beta2)),
                  'epsilon': self.epsilon,
                  'schedule_decay': float(K.get_value(self.schedule_decay))}
        base_config = super(Nadam, self).get_config()
        return dict(list(base_config.items()) + list(config.items()))

keras.optimizers.Nadam = Nadam

[sandeep-krishnamurthy]

Thanks @laucheukhim - We will try to use this patch in Keras 2 work we are currently working on at - https://github.com/deep-learning-tools/keras/tree/keras2_mxnet_backend

[alexrisman]

@laucheukhim: Worked like a charm, thanks so much!