CNN-LSTM model

[nidhijoshi123]

" I extract the activations of the CNN manually, then use the activations as inputs for training a separate LSTM ".

When you say 'extract the activations' , do you mean extract the predicted values from the CNN layer output and feed them as inputs to LSTM ? Conceptually I wanted to know if the input to the LSTM model will just be the predicted labels from CNN model. Can I use one-hot encoded labels ? Also, I was unable to find this piece of code in your Github repository. It would be very hepful if you can point me to the python class which feeds the CNN activations into LSTM.

[skjerns]

I am using the activations of the fully connected layer before the softmax, which has a latent representation of the input signal (eg 1500 values), see this figure

The code of this is a bit hidden in keras_utils.py’ within the function ‘cv’ I am calling ‘

extracted = get_activations(model, train_data + val_data + test_data, lname, batch_size*2, cropsize=cropsize)

, with ‘lname’ being the name of the last or second last fully connected layer. The get activations function is also within the same file. By the way I’m sure there are more elegant ways to code this by now, with the new keras API.

[nidhijoshi123]

Hi Simon, Thanks for your prompt reply. One small confusion: Isn't the following the line that extracts features from CNN..

*features = get_activations(model, train_data + val_data, lname, batch_size2, cropsize=cropsize)**

This line was found in keras_utils.py, train_models (LSTM training part).

Kind regards, Nidhi Joshi

[skjerns]

It is, exactly!