Topic classification model in keras with MXNet backend errors out - Too many slices such that some splits are empty

[sandeep-krishnamurthy]

Running Reuters Topic Classification Model under keras/example for MXNet backend with 8 GPU errors out with following error: Error: Too many slices such that some splits are empty

Code - https://github.com/fchollet/keras/blob/master/examples/reuters_mlp.py

Setting:

1. MXNet latest commit till (March 8, 2017)
2. dmlc/keras latest commit till (March 8, 2017)
3. Number of GPU - 8

However, same example works with Tensorflow backend.

Also, with MXNet backend for 1, 2, 4 GPUs it works fine.

[sandeep-krishnamurthy]

Stack trace for reference:

Loading data... 8982 train sequences 2246 test sequences 46 classes Vectorizing sequence data... X_train shape: (8982, 1000) X_test shape: (2246, 1000) Convert class vector to binary class matrix (for use with categorical_crossentropy) Y_train shape: (8982, 46) Y_test shape: (2246, 46) Building model... Train on 8083 samples, validate on 899 samples Epoch 1/5 /usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/bucketing_module.py:348: UserWarning: Optimizer created manually outside Module but rescale_grad is not normalized to 1.0/batch_size/num_workers (1.0 vs. 0.03125). Is this intended? force_init=force_init) 8000/8083 [============================>.] - ETA: 0s - loss: 3.6955 - acc: 0.3495Traceback (most recent call last): File "reuters_mlp.py", line 64, in verbose=1, validation_split=0.1) File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/models.py", line 672, in fit initial_epoch=initial_epoch) File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/engine/training.py", line 1198, in fit initial_epoch=initial_epoch) File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/engine/training.py", line 907, in _fit_loop verbose=0) File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/engine/training.py", line 992, in _test_loop batch_outs = f(ins_batch) File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/engine/training.py", line 1968, in testfunction data, label, , data_shapes, label_shapes = self._adjust_module(inputs, 'test') File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/engine/training.py", line 1914, in _adjust_module self._mod._curr_module.reshape(data_shapes, label_shapes) File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/module.py", line 427, in reshape self._exec_group.reshape(self._data_shapes, self._label_shapes) File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/executor_group.py", line 311, in reshape self.bind_exec(data_shapes, label_shapes, reshape=True) File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/executor_group.py", line 276, in bind_exec self.data_layouts = self.decide_slices(data_shapes) File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/executor_group.py", line 221, in decide_slices self.slices = _split_input_slice(self.batch_size, self.workload) File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/executor_manager.py", line 48, in _split_input_slice raise ValueError('Too many slices such that some splits are empty') ValueError: Too many slices such that some splits are empty

[imranshaikmuma]

increase batch_size