can you give me an example for cnn connect with LSTM

[1461190388]

@anayebi

[1461190388]

i have create a network but it don‘t work and have many errors

[anayebi]

Happy to help! Usually, I connect the two using my TimeDistributedConvolution2D layer in keras.layers.extra rather than the Convolution2D layer. So, here is an example (note that the layers that I use in this example which are only in keras.layers.extra are TimeDistributedConvolution2D, TimeDistributedMaxPooling2D, and TimeDistributedFlatten):

model.add(TimeDistributedConvolution2D(16, 40, 9, 9, border_mode='full')) model.add(Activation('relu')) model.add(TimeDistributedMaxPooling2D(poolsize=(2, 2), ignore_border=True)) model.add(TimeDistributedFlatten()) model.add(TimeDistributedDense(6400, 32)) model.add(Activation('relu')) model.add(LSTM(32, 32, return_sequences=True))

So basically, this is a convolutional network followed by max pooling (which also has to be distributed across time), followed by flattening (again across time), which then gets fed into a dense layer that feeds to an LSTM.

[1461190388]

Thank for your help! when i construct my network,it have many error.i don't know why. Next i will give you my network. model = Sequential() model.add(TimeDistributedConvolution2D(10, 3, 17,11, border_mode='full',input_shape=(time_steps,nchannels,img_rows,img_cols))) # note that input_shape must be given in the latest keras model.add(Activation('sigmoid'))

model.add(TimeDistributedConvolution2D(10, 32, 9, 7)) model.add(Activation('sigmoid')) model.add(TimeDistributedMaxPooling2D(pool_size=(2, 2))) model.add(TimeDistributedFlatten()) model.add(TimeDistributedDense(512)) model.add(Activation('sigmoid')) model.add(Dropout(0.5)) model.add(LSTM(512, return_sequences=True)) # output shape: (nb_samples, timesteps, *) model.add(Flatten()) model.add(Dense(1000)) model.add(Dropout(0.5)) model.add(Dense(100)) model.add(Dropout(0.5)) model.add(Dense(2)) print 'build model===========' model.compile(loss='categorical_crossentropy', optimizer='adadelta') print("Train...") model.fit(train_data, y_train, batch_size=batch_size, nb_epoch=2, validation_data=(test_data, y_test), show_accuracy=True) #note train_data’s Input shape: (num_samples, num_timesteps, stack_size, num_rows, num_cols),y_train is label,y_train's shape :(num_samples,label) score, acc = model.evaluate(test_data, y_test, batch_size=batch_size, show_accuracy=True) #note test_data is the same as train_data,y_test is the same as y_test. print('Test score:', score) print('Test accuracy:', acc)

could you give me some advice？ Thank you for very much.

[anayebi]

I think the issue is that Keras recently introduced a new API (announced here: https://groups.google.com/forum/m/#!topic/keras-users/iWVrWpR_eaQ), which was not the case when I wrote these layers.

As a result, in my layers, you do not need to specify an input shape. What happens if you change the second line of your code:

model.add(TimeDistributedConvolution2D(10, 3, 17,11, border_mode='full',input_shape=(time_steps,nchannels,img_rows,img_cols)))

to not include the input shape:

model.add(TimeDistributedConvolution2D(10, 3, 17,11, border_mode='full')

Does it still work?

If it doesn't work let me know, and I will update my layers to match the current Keras API. On the other hand, if you would rather not wait, then try checking out a version of Keras prior to October 6th (which was around when the new API was introduced). Sorry for the inconvenience, and thank you for bringing the API change in Keras to my attention!

[1461190388]

thanks very much for your reply,I know your codes depends on old keras version so i have changed your code to follow the latest keras version ,but it did't work .i don't know why. Next i will use your extra.py and match with a version of Keras prior to October 6th.thank you for your advice again.

[anayebi]

I just pushed the newest version of the layers to now work with the current Keras API. As a note, just with the Convolution2D layers in Keras, if TimeDistributedConvolution2D is being used as the first layer of the network, you must specify the argument, input_shape=(num_time_steps, num_channels, num_rows, num_cols).

[NastaranMrad]

First of all thanks for this nice package. Then I have a question regard using keras-extra with tensorflow backend. For some reasons I must use the "same" method for padding in convolution and it is not supported in Theano (my code is running perfectly with Theano backend and valid border mode). This is why I must use the Tensorflow backend. I have reshaped my data based on the data structure in tensor flow (samples_timesteps_rows_columns_channels) and I passed the batch_input_shape parameter to the first conv layer. But I receive "AssertionError" in line 128 of "recurrent.py" function. The error happens after flattening and in the LSTM layer. It seems the output of flattening layer is not 3 dimensional which evokes the assertion error! I am using Keras 3.1 in my code. Can you please help me on this issue? Thank you very much in advance.

[anayebi]

@NastaranMrad Try using Keras 3.0, since that is the latest version of Keras that this package supports.

[NastaranMrad]

@anayebi Thanks for the answer. I have changed my keras to 3.0 (I have got this version from https://github.com/fchollet/keras/releases/tag/0.3.0). Now I receive the following error: "AssertionError: Keyword argument not understood: batch_input_shape" The error is in core.py line 23. I think I need to have exactly the Keras version you built the kera-extra upon. Can you please send me the link? Thank you in advance

[mlopezm]

Look at this blog https://yerevann.github.io/2016/06/26/combining-cnn-and-rnn-for-spoken-language-identification/ It is incredibly well explained