Kernel size (3) exceeds input (1 padded to 1)

[NTNguyen13]

I just started using MXNet as Backend for Keras, I have followed the guide as follow: 1, Edit the keras.json file:

{
    "image_data_format": "channels_first",
    "epsilon": 1e-07,
    "floatx": "float32",
    "backend": "mxnet"
}

2, Change the input method of my model: input = Input(shape=(img_channels, img_height, img_width))

But when I tried to stack Conv2D, I still got the warning message:

UserWarning: MXNet Backend performs best with `channels_first` format. Using `channels_last` will significantly reduce performance due to the Transpose operations. For performance improvement, please use this API`keras.utils.to_channels_first(x_input)`to transform `channels_last` data to `channels_first` format and also please change the `image_data_format` in `keras.json` to `channels_first`.Note: `x_input` is a Numpy tensor or a list of Numpy tensorRefer to: https://github.com/awslabs/keras-apache-mxnet/tree/master/docs/mxnet_backend/performance_guide.md
  train_symbol = func(*args, **kwargs)

And when I tried to stack more layers, I got this errror:

  File "/home/cngc3/anaconda3/envs/mxnet-gpu/lib/python3.5/site-packages/mxnet/base.py", line 149, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator max_pooling2d_6/pool2d1: [14:16:41] src/operator/nn/pooling.cc:141: Check failed: param.kernel[0] <= dshape[2] + 2 * param.pad[0] kernel size (3) exceeds input (1 padded to 1)

If I use this input method: input = Input(shape=(img_height, img_width, img_channels))

I still got the UserWarning message, but the network can train without any problem, although it is much slower.

Thank you for your help

[roywei]

@NTNguyen13 thanks for the issue, we need for information on how you stacked Conv2D? A minimum reproducible code could help a lot.

[NTNguyen13]

#model
input = Input(shape=(3, 250, 250))
#x = Lambda(filter_layer)(input)
x = Conv2D(128, (4, 4), padding='same')(input)
x = ELU(alpha=1.0)(x)
x = BatchNormalization(axis=1)(x)

x = Conv2D(128, (3, 3), padding="same")(x)
x = ELU(alpha=1.0)(x)
x = BatchNormalization(axis=1)(x)
x = MaxPooling2D(pool_size=(3, 3))(x)

x = Conv2D(128, (5, 5), padding="same")(x)
x = ELU(alpha=1.0)(x)
x = BatchNormalization(axis=1)(x)
x = MaxPooling2D(pool_size=(3, 3))(x)

This is the maximum of layers I can stack until I got the error. Normally when using Tensorflow backend I can stack at least 2 more blocks of Conv2D and MaxPooling

In this case, the error is:

line 149, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator max_pooling2d_17/pool2d1: [17:51:45] src/operator/nn/pooling.cc:141: Check failed: param.kernel[0] <= dshape[2] + 2 * param.pad[0] kernel size (3) exceeds input (1 padded to 1)

[roywei]

@NTNguyen13 Hi, I was still not able to reproduce your error. I created some random data and tried to fit your model, it was working fine. Could you provide more details? what other layers you used and what's the data like. Also what's the version you are using? (try pip list | grep mxnet to check), you can upgrade to latest master by pip install --upgrade keras-mxnet --pre and pip install --upgrade mxnet --pre

import numpy as np
from keras.layers import Input, Conv2D, ELU, BatchNormalization, MaxPooling2D, Dense, Flatten
from keras.models import Model
from keras import backend as K
x_train = np.random.randint(0,256, (1000, 3, 250, 250))
y_train =   np.random.randint(0,10, (1000,))
input = Input(shape=(3, 250, 250))
#x = Lambda(filter_layer)(input)
x = Conv2D(128, (4, 4), padding='same')(input)
x = ELU(alpha=1.0)(x)
x = BatchNormalization(axis=1)(x)

x = Conv2D(128, (3, 3), padding="same")(x)
x = ELU(alpha=1.0)(x)
x = BatchNormalization(axis=1)(x)
x = MaxPooling2D(pool_size=(3, 3))(x)

x = Conv2D(128, (5, 5), padding="same")(x)
x = ELU(alpha=1.0)(x)
x = BatchNormalization(axis=1)(x)
x = MaxPooling2D(pool_size=(3, 3))(x)
x = Flatten()(x)
output = Dense(10, activation='softmax')(x)
model = Model(input, output)
model.compile(optimizer='sgd', loss='sparse_categorical_crossentropy')
model.fit(x_train,y_train)

output:

/Users/lawei/anaconda3/bin/python /Users/lawei/Documents/Workspace/roywei/keras-apache-mxnet/keras/backend/test2.py
Using MXNet backend
/Users/lawei/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Epoch 1/1
/Users/lawei/anaconda3/lib/python3.6/site-packages/mxnet/module/bucketing_module.py:408: UserWarning: Optimizer created manually outside Module but rescale_grad is not normalized to 1.0/batch_size/num_workers (1.0 vs. 0.03125). Is this intended?
  force_init=force_init)
[11:58:28] src/operator/nn/mkldnn/mkldnn_base.cc:74: Allocate 1024000000 bytes with malloc directly
[11:58:28] src/operator/nn/mkldnn/mkldnn_base.cc:74: Allocate 589824 bytes with malloc directly

  32/1000 [..............................] - ETA: 12:39 - loss: 3.3742

[NTNguyen13]

@roywei I'm using keras-mxnet==2.2.2 and mxnet-cu90mkl==1.2.1.post1

Edit: I solved the problem by updating mxnet-cu90mkl to the latest version. However I got another issue in multi gpu, I will open another issue