sequence classification using LSTM

[charles20140410]

I do sequence classification using LSTM and The length of sequences are variable. So I need to pad it with max-length, and I also set mask_zero flag = true.

My Network model like below:

http://deeplearning.net/tutorial/lstm.html

My code as below:

def LSTM_Classification():

max_features = 19694+1
batch_size = 32
embedding_dims = 32
nb_epoch = 3
path = "./dataset/SemEval_task3.pkl"
max_words = max_features

(X_train, y_train), (X_test, y_test) = guan_SemEval_task3.load_data(path)
x_new_train = padding_all(X_train)
y_new_train = np.array([(1 if y is 0 else 0, 1 if y is 1 else 0, 1 if y is 2 else 0) for y in y_train])

x_new_test = padding_all(X_test)    
y_new_test = np.array([(1 if y is 0 else 0, 1 if y is 1 else 0, 1 if y is 2 else 0) for y in y_test])
print (x_new_train.shape)
print (y_new_train.shape)
print (x_new_test.shape)
print (y_new_test.shape)

print('Build model...')
model = Sequential()
model.add(Embedding(max_features, embedding_dims, dropout=0.2,mask_zero=True))
model.add(LSTM(32, dropout_W=0.2, dropout_U=0.2,return_sequences=True))  
print(model.output_shape) #(None, None, 32)

model.add(TimeDistributed(Dense(1)))
print(model.output_shape) #(None, None, 1)

avgpool = Lambda(lambda x: K.mean(x, axis=1, keepdims=False), output_shape=lambda x: (x[0], x[2]))
avgpool.supports_masking = True
model.add(avgpool)    
print(model.output_shape) #(None, 1)

#model.add(Dense(3))
dense3 = Dense(3)
dense3.supports_masking = True
model.add(dense3)
print(model.output_shape) #(None, 1)
model.add(Activation('softmax'))
print(model.output_shape) #(None, 1)

model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])
model.summary()
early_stop = EarlyStopping(monitor='val_loss', patience=20, verbose=1)
model.fit(x_new_train, y_new_train,nb_epoch=nb_epoch,batch_size=batch_size,
          validation_data=(x_new_test, y_new_test),callbacks=[early_stop])    

print('Testing...')
score, acc = model.evaluate(x_new_test, y_new_test,batch_size=batch_size)
print('Test score:', score)
print('Test accuracy:', acc)

the output_shape of the model seem to be right. but when I run this code, the error message as below:

(16541L, 849L) (16541L, 3L) (1976L, 849L) (1976L, 3L) Build model... (None, None, 32) (None, None, 1) (None, 1) (None, 3) (None, 3)

---

Layer (type) Output Shape Param # Connected to

embedding_5 (Embedding) (None, None, 32) 630240 embedding_input_5[0][0]

---

lstm_5 (LSTM) (None, None, 32) 8320 embedding_5[0][0]

---

timedistributed_5 (TimeDistribute(None, None, 1) 33 lstm_5[0][0]

---

lambda_5 (Lambda) (None, 1) 0 timedistributed_5[0][0]

---

dense_10 (Dense) (None, 3) 6 lambda_5[0][0]

---

activation_3 (Activation) (None, 3) 0 dense_10[0][0]

Total params: 638599

---

Train... Train on 16541 samples, validate on 1976 samples Epoch 1/3 Traceback (most recent call last):

File "", line 1, in runfile('D:/prj_python_spyder/test1/test1/programs/test_fasttext.py', wdir='D:/prj_python_spyder/test1/test1/programs')

File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 714, in runfile execfile(filename, namespace)

File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile exec(compile(scripttext, filename, 'exec'), glob, loc)

File "D:/prj_python_spyder/test1/test1/programs/test_fasttext.py", line 508, in main();

File "D:/prj_python_spyder/test1/test1/programs/test_fasttext.py", line 498, in main doLSTM_AVGPooling()

File "D:/prj_python_spyder/test1/test1/programs/test_fasttext.py", line 305, in doLSTM_AVGPooling validation_data=(x_new_test, y_new_test),callbacks=[early_stop])

File "C:\Anaconda2\lib\site-packages\keras\models.py", line 597, in fit sample_weight=sample_weight)

File "C:\Anaconda2\lib\site-packages\keras\engine\training.py", line 1107, in fit callback_metrics=callback_metrics)

File "C:\Anaconda2\lib\site-packages\keras\engine\training.py", line 825, in _fit_loop outs = f(ins_batch)

File "C:\Anaconda2\lib\site-packages\keras\backend\theano_backend.py", line 657, in call return self.function(*inputs)

File "C:\Anaconda2\lib\site-packages\theano\compile\function_module.py", line 879, in call storage_map=getattr(self.fn, 'storage_map', None))

File "C:\Anaconda2\lib\site-packages\theano\gof\link.py", line 325, in raise_with_op reraise(exc_type, exc_value, exc_trace)

File "C:\Anaconda2\lib\site-packages\theano\compile\function_module.py", line 866, in call self.fn() if output_subset is None else\

ValueError: Input dimension mis-match. (input[0].shape[1] = 32, input[1].shape[1] = 849) Apply node that caused the error: Elemwise{mul,no_inplace}(InplaceDimShuffle{x,0}.0, Elemwise{Cast{float32}}.0) Toposort index: 350 Inputs types: [TensorType(float32, row), TensorType(float32, matrix)] Inputs shapes: [(1L, 32L), (32L, 849L)] Inputs strides: [(128L, 4L), (3396L, 4L)] Inputs values: ['not shown', 'not shown'] Outputs clients: [[Sum{axis=[1], acc_dtype=float64}(Elemwise{mul,no_inplace}.0)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'. HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

The 849 of the max length I pad in sequences.

Could someone please help to fix this error ? I have got stuck several days. Thanks a lot.