Requirement failed: element number must match Reshape size. But In Reshapef98c1244 : element number is: 4224 , reshape size is: 1408

[m-developer96]

Hi everyone. I used this tutorial to build a simple text classifier. I used Sentiment-140 dataset and FastText pre-trained word embeddings for this model and just change below variables: embedding_dim = 100 sequence_len = 100 But my model doesn't run successfully and I faced with this error :

`An error occurred while calling o346.optimize.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 445.0 failed 4 times, most recent failure: Lost task 1.3 in stage 445.0 (TID 1338, 172.18.16.52, executor 0): Layer info: Sequential[673284c4]{

  [input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> output]

  (1): Reshape[b618f1d0](100x1x100)

  (2): SpatialConvolution[conv1](100 -> 128, 5 x 1, 1, 1, 0, 0)

  (3): ReLU[fb58982a](0.0, 0.0)

  (4): SpatialMaxPooling[933cd7b7](5, 1, 5, 1, 0, 0)

  (5): SpatialConvolution[conv2](128 -> 128, 5 x 1, 1, 1, 0, 0)

  (6): ReLU[20fa73c0](0.0, 0.0)

  (7): SpatialMaxPooling[1da4f934](5, 1, 5, 1, 0, 0)

  (8): Reshape[f98c1244](128)

  (9): Linear[fc1](128 -> 100)

  (10): Linear[fc2](100 -> 2)

  (11): LogSoftMax[97739e2a]

}/Reshape[f98c1244](128)

java.lang.IllegalArgumentException: requirement failed: element number must match Reshape size. But In Reshapef98c1244 : element number is: 4224 , reshape size is: 1408`

How can I fix this error? Help me plz. Thanks :)

[Le-Zheng]

Perhaps there might be an error if your data is not reshaped as the same size as the example.

[jason-dai]

@hkvision please take a look.

[hkvision]

Hi @m-developer96,

The error is due to you have different embedding dim and sequence length as the example you refer to, you need to change the model structure accordingly.

To be more specific, when the embedding dim and sequence length are both 50 in the example we give,

model = Sequential()
model.add(Reshape([embedding_dim, 1, sequence_len]))
model.add(SpatialConvolution(embedding_dim, 128, 5, 1).set_name('conv1'))
model.add(ReLU())
model.add(SpatialMaxPooling(5, 1, 5, 1))
model.add(SpatialConvolution(128, 128, 5, 1).set_name('conv2'))
model.add(ReLU())
model.add(SpatialMaxPooling(5, 1, 5, 1))

The model output before the Reshape([128]) layer is of shape (batch, 128, 1, 1) and therefore you can do the reshape to let the output become (batch, 128)

However, if you change your embedding dim and sequence length to 100, the model output before the Reshape([128]) layer becomes of the shape (batch, 128, 1, 3), and therefore you can't reshape it to (batch, 128) and that's why you get the error.

To fix this, one simplest choice is to change the Reshape layer to Reshape([128 * 3]), but I don't think this is the best choice for such a network architecture. Probably you need to change the parameters of the convolution layers to make the output before the reshape layer have the shape (batch, 128, 1, 1).

[hkvision]

Hi @m-developer96,

Also as you may see, using the old interface of BigDL, you need to calculate the output shape of every layer in the model carefully to avoid certain errors. Thus, if you wish, you are recommended to use the Keras-Style APIs here for model definition: https://github.com/intel-analytics/analytics-zoo/tree/master/pyzoo/zoo/pipeline/api/keras , which supports shape inference and prevents you constructing a shape-incompatible model before you train it.

BTW, the example you are referring to is BigDL 0.1, which is probably a bit too old. We have newer examples for you to have a reference: https://github.com/intel-analytics/analytics-zoo/tree/master/pyzoo/zoo/examples/textclassification https://github.com/intel-analytics/analytics-zoo/tree/master/apps/sentiment-analysis

Enjoy your coding and feel free to tell us if you have further problems.

[m-developer96]

hi @hkvision Thank you so much for your help.

[hkvision]

@m-developer96 You are welcome. I will close this issue first. Feel free to tell us if you have further issues.