Convolutional RNNs for 1D sequences

[iskandr]

I would like to user multiple scales of convolutions over character sequences, followed by RNN layers. This is currently clumsily possible by ignoring masking but I would love if there were something like ConvLSTM1D/ConvGRU1D which enabled this use-case directly.

Thanks!

[theotheo]

Sorry, but what do you mean by words "ignoring masking"?

[iskandr]

I pad shorter sequences with 0 at the end and then embed that padding token along with other "real" tokens (i.e. mask_zero=False). Does that make sense?

[fchollet]

You mean, a ConvLSTM1D layer that would support masking?

[iskandr]

@fchollet I might be misunderstanding how the convolutional recurrent models work, but don't they already support masking?

My use-case is biological sequences, so the dimensionality is typically (n_timesteps, embedding_dims). My loose understanding of ConvLSTM2D is that it's meant for video data with dimensions like (n_timesteps, width, height, channels) Is there a way to use ConvLSTM2D for my case? Alternatively, would it be possible to make Conv1D work with masked sequences of varying lengths?

[fchollet]

Alternatively, would it be possible to make Conv1D work with masked sequences of varying lengths?

That would be a nice feature to have, but not easy to implement. Feel free to look into it, or find people to look into it.

Is there a way to use ConvLSTM2D for my case?

Why not feed your data as (n_timesteps, length, 1, channels)?

[iskandr]

Why not feed your data as (n_timesteps, length, 1, channels)

Since I just have a single sequence I would have a dataset with dimensions (length, 1, 1, channels) or alternatively, (1, length, 1, channels) -- would that work with ConvLSTM2D? (I suspect not)

[fchollet]

If you do not have a spatial component, I don't understand why you would want to use ConvLSTM?

[iskandr]

I want temporal convolutional filtering, with an RNN to reason about relative order of filter responses. I don't know if ConvLSTM is the right tool for this -- it seems like a mask-aware Conv1D would be better but I understand that might be a significant undertaking.

[fchollet]

It seems like ConvRNN is not what you need in your case.

I want temporal convolutional filtering, with an RNN to reason about relative order of filter responses

It seems like you are looking for a stack of Conv1D layers followed by a RNN layer?

[iskandr]

Yep, that's exactly what I want. The problem is, as far as I can tell, there is no Conv1D which works with variable length sequences (by propagating a mask up to the RNN layer).

On Apr 9, 2017 4:33 PM, "François Chollet" notifications@github.com wrote:

It seems like ConvRNN is not what you need in your case.

I want temporal convolutional filtering, with an RNN to reason about relative order of filter responses

It seems like you are looking for a stack of Conv1D layers followed by a RNN layer?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/fchollet/keras/issues/6150#issuecomment-292814555, or mute the thread https://github.com/notifications/unsubscribe-auth/AAC9OW1u05ylHDiVpCSMcqqYO5RBqnFnks5ruU6MgaJpZM4MzTBI .

[fchollet]

If you know how the output of your Conv1D should be masked, you can pass an explicit mask argument (tensor) to your RNN layer.

That, or add a compute_output_mask method to conv layers.

[ja-94]

@iskandr How did you end up solving the problem? I have a very similar situation, and I'm using ConvLSTM2D with kernel_size=(1,1) and input shape=(timesteps, channels, 1, 1) @fchollet The network is getting very good accuracies so it surely works, but maybe I should be using something else? Conv1D/SeparableConv1D?

[aviogit]

@ja-94 this is great news to hear, can you post your model? I'm a total beginner with neural networks (tried just a few simple LSTM/GRU models)... I was wondering if ConvLSTM2D were good for acoustic data and your answer seems to definitely point toward the "yes". Now the problem is building the model! :)

[ja-94]

@ja-94 this is great news to hear, can you post your model? I'm a total beginner with neural networks (tried just a few simple LSTM/GRU models)... I was wondering if ConvLSTM2D were good for acoustic data and your answer seems to definitely point toward the "yes". Now the problem is building the model! :)

#################################################  CREATE MODEL  ###############################################################################
    maxlen=x_train.shape[1]
    input_dim=x_train.shape[2]
    model = Sequential()
    input_shape=(maxlen,input_dim,1,1)
    print('input_shape=', input_shape)
    model.add(ConvLSTM2D(12, data_format='channels_last', 
                            input_shape=input_shape, 
                            kernel_size=(1, 1), 
                            strides=(1, 1), 
                            padding='valid', 
                            dilation_rate=(1, 1), 
                            activation='tanh', 
                            recurrent_activation='hard_sigmoid', 
                            use_bias=True, 
                            kernel_initializer='glorot_uniform', 
                            recurrent_initializer='orthogonal', 
                            bias_initializer='zeros', 
                            unit_forget_bias=True, 
                            kernel_regularizer=None, 
                            recurrent_regularizer=None, 
                            bias_regularizer=None, 
                            activity_regularizer=None, 
                            kernel_constraint=None, 
                            recurrent_constraint=None, 
                            bias_constraint=None, 
                            return_sequences=True, 
                            go_backwards=False, 
                            stateful=False, 
                            dropout=0.0, 
                            recurrent_dropout=0.0))
    model.add(Flatten())
    model.add(Dense(y_train.shape[1], activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adadelta', metrics=['accuracy', metrics.binary_accuracy])

    model.summary()