Convolutional1D for time series data

[ytsaig]

I am trying to build a convolutional network to classify 1D data (time series data). Specifically, the input consists of vectors of length 256. I tried using the Convolutional1D layer to construct the network, but ran into trouble with specifying the input dimensions.

I consulted the docs, which state that:

When using this layer (Convolutional1D) as the first layer in a model, either provide the keyword argument input_dim (int, e.g. 128 for sequences of 128-dimensional vectors), or input_shape (tuple of integers, e.g. (10, 128) for sequences of 10 vectors of 128-dimensional vectors).

I opted for specifying the input dimensions via input_dim=256. Here's the full code to construct the network, using a single convolutional layer:

from keras.models import Sequential
from keras.layers.core import Dense, Dropout, Activation, Flatten
from keras.layers.convolutional import Convolution1D, MaxPooling1D

m2 = Sequential()
m2.add(Convolution1D(nb_filter=nb_filter,
                        filter_length=filter_length,
                        border_mode="valid",
                        activation="relu", 
                        input_dim=256))
m2.add(MaxPooling1D(pool_length=2))
m2.add(Flatten())
m2.add(Dense(128))
m2.add(Dropout(0.25))
m2.add(Activation('relu'))

The code fails on the line m2.add(Dense(128)) with the message

TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

Is there something in the docs that I am missing?

[ytsaig]

After looking at Issue #529, it seems that I may have misinterpreted the terminology of the various dimensions. Looks like steps in fact refers to the length of the time series, and the dimension of the vector refers to the number of features per time point, perhaps similar to channels in a 2D image? Not sure, the terminology is a bit cryptic.

So I tried initializing the Convolutional1D layer with input_shape=(256, 1) instead, and indeed the model compiled fine. However, when I try to fit the model using

model.fit(X_train, Y_train, batch_size=128, nb_epoch=100)

where X_train is a numpy array with dimensions (n_samples, 256), I get the following error

TypeError: ('Bad input argument to theano function with name "/home/ytsaig/packages/anaconda/lib/python2.7/site-packages/keras/models.py:401" at index 0(0-based)', 'Wrong number of dimensions: expected 3, got 2 with shape (128, 256).')

What is the appropriate input format that the network will accept?

[darzok]

hello,

I'm trying to do axactly the same thing (with forex data). Actually I am a student in Mathematic field so even if I know how programming I am not a programmer.

to start I tryed to copy / past the 'imbd_cnn' example but I have an error :

Traceback (most recent call last): File "/home/benjamin/Bureau/test_MNIST/test classe (copie)/CNN_1D.py", line 55, in subsample_length=1)) TypeError: init() takes at least 4 arguments (6 given)

Does anybody know why ?

[darzok]

After adapting the ytsaig code to my task I have the same error :

TypeError: ('Bad input argument to theano function with name "/usr/local/lib/python2.7/dist-packages/keras/models.py:402" at index 0(0-based)', 'Wrong number of dimensions: expected 3, got 2 with shape (15, 60).')

my code :

sizeTraining=0.9
sizeTest=0.1
window=60
h=10

print('Loading data ...')
forex = FOREX()
(X_train, y_train),(X_test,y_test) = forex.load_data(sizeTraining, sizeTest, window, h)

X_train = X_train*10000
y_train = y_train*10000

X_test = X_test*10000
y_test = y_test*10000

print(X_train.shape[0],'train samples')
print(X_test.shape[0],'test samples')

nb_filter = 3
filter_length = 6

model = Sequential()

model.add(Convolution1D(nb_filter=nb_filter,
                        filter_length=filter_length,
                        border_mode="valid",
                        activation="relu", 
                        input_dim=window))
model.add(MaxPooling1D(pool_length=2))
model.add(Dropout(0.25))

model.add(Flatten())

model.add(Dense(20,300))
model.add(Dropout(0.25))
model.add(Activation('relu'))

model.add(Dense(128, 1))
model.add(Activation('linear'))

model.compile(loss='mean_squared_error', optimizer='adadelta')

model.fit(X_train, y_train, batch_size=15, nb_epoch=30, show_accuracy=True, validation_data=(X_test, y_test))

[dbonadiman]

This code do not work with the newer keras implementation, you should do something like that: Check what version you have installed (in the case i suggest to update it).

I don't know the dataset you are using but i hope this can help you.

sizeTraining=0.9
sizeTest=0.1
window=60
h=10

print('Loading data ...')
forex = FOREX()
(X_train, y_train),(X_test,y_test) = forex.load_data(sizeTraining, sizeTest, window, h)

X_train = X_train*10000
y_train = y_train*10000

X_test = X_test*10000
y_test = y_test*10000

print(X_train.shape[0],'train samples')
print(X_test.shape[0],'test samples')

nb_filter = 3
filter_length = 6

model = Sequential()

model.add(Convolution1D(nb_filter=nb_filter,
                        filter_length=filter_length,
                        border_mode="valid",
                        activation="relu", 
                        input_shape=(window, 1)))
model.add(MaxPooling1D(pool_length=2))
model.add(Dropout(0.25))

model.add(Flatten())

model.add(Dense(300))
model.add(Dropout(0.25))
model.add(Activation('relu'))

model.add(Dense(1))
model.add(Activation('linear'))

model.compile(loss='mean_squared_error', optimizer='adadelta')

model.fit(X_train, y_train, batch_size=15, nb_epoch=30, show_accuracy=True, validation_data=(X_test, y_test))

If you add more information from your error we can figure out better what is causing it.

[darzok]

hi ,

First of all thank you very much for your answer. I use a personnal dataset.

I change my code with yours and I had this error :

Traceback (most recent call last):
  File "/home/benjamin/Bureau/test_MNIST/test classe (copie)/main.py", line 42, in <module>
    input_shape=(window, 1)))
TypeError: __init__() got an unexpected keyword argument 'input_shape'

so I opened a terminal and I put :

pip freeze

I had theano 0.7.0 and Keras 0.1.3

then I wanted to uptade keras and I follow the installation instruction

git clone https://github.com/fchollet/keras.git
cd keras
sudo python setup.py install
sudo pip install keras

so now I have keras in 0.2.0. I tryed to ran my code, then when I compile my network I had this message :

Traceback (most recent call last):
  File "/home/benjamin/Bureau/test_MNIST/test classe (copie)/main.py", line 56, in <module>
    model.compile(loss='mean_squared_error', optimizer='adadelta')
  File "build/bdist.linux-x86_64/egg/keras/models.py", line 353, in compile
    self.y_train = self.get_output(train=True)
  File "build/bdist.linux-x86_64/egg/keras/layers/containers.py", line 52, in get_output
    return self.layers[-1].get_output(train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 412, in get_output
    X = self.get_input(train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 89, in get_input
    return self.previous.get_output(train=train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 586, in get_output
    X = self.get_input(train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 89, in get_input
    return self.previous.get_output(train=train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 412, in get_output
    X = self.get_input(train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 89, in get_input
    return self.previous.get_output(train=train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 385, in get_output
    X = self.get_input(train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 89, in get_input
    return self.previous.get_output(train=train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 586, in get_output
    X = self.get_input(train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 89, in get_input
    return self.previous.get_output(train=train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 492, in get_output
    X = self.get_input(train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 89, in get_input
    return self.previous.get_output(train=train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 385, in get_output
    X = self.get_input(train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 89, in get_input
    return self.previous.get_output(train=train)
  File "build/bdist.linux-x86_64/egg/keras/layers/convolutional.py", line 310, in get_output
    X = self.get_input(train)
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 89, in get_input
    return self.previous.get_output(train=train)
  File "build/bdist.linux-x86_64/egg/keras/layers/convolutional.py", line 150, in get_output
    output = self.activation(conv_out + self.b.dimshuffle('x', 0, 'x', 'x'))
  File "build/bdist.linux-x86_64/egg/keras/activations.py", line 20, in relu
    return T.nnet.relu(x)
AttributeError: 'module' object has no attribute 'relu'

so I tryed to remove 'relu' in conv1D layer like that :

model.add(Convolution1D(nb_filter=nb_filter,
                        filter_length=filter_length,
                        border_mode="valid", 
                        input_shape=(window, 1)))

but I have still the error .... do you have an idea ? or anybody else ?

[dbonadiman]

I rarely use convolution at the first layer so probably i suggested it wrong, if i can object something in general about keras is that there are inconsistencies on the shape definition on the various layers, however the input shape is actually not a parameter for the convolution1d but it is defined internally as a combination of input_lenght and input_dim try with the former and it should work.

[darzok]

thank you for your reply, I think by "the former" you mean my first code, right ? (because google trad say that it means the "old one" )

so I tryed with my first code this one :

sizeTraining=0.9 #size of training set in percent of data set size = nb of samples for training set
sizeTest=0.1 #size of set test in percent of data set size = nb of samples for test set
window=60 # size of one sample
h=10 #prediction = the last value of one sample + h 

print('Loading data ...')
forex = FOREX()
(X_train, y_train),(X_test,y_test) = forex.load_data(sizeTraining, sizeTest, window, h)

X_train = X_train*10000
y_train = y_train*10000

X_test = X_test*10000
y_test = y_test*10000

print(X_train.shape[0],'train samples')
print(X_test.shape[0],'test samples')

nb_filter = 3
filter_length = 6

model = Sequential()

model.add(Convolution1D(nb_filter=nb_filter,
                        filter_length=filter_length,
                        border_mode="valid",
                        activation="relu", 
                        input_dim=window))
model.add(MaxPooling1D(pool_length=2))
model.add(Dropout(0.25))

model.add(Flatten())

model.add(Dense(18,300))
model.add(Dropout(0.25))
model.add(Activation('relu'))

model.add(Dense(128, 1))
model.add(Activation('linear'))

model.compile(loss='mean_squared_error', optimizer='adadelta')

model.fit(X_train, y_train, batch_size=15, nb_epoch=30, show_accuracy=True, validation_data=(X_test, y_test))

then I have this error now :

Traceback (most recent call last):
  File "/home/benjamin/Bureau/test_MNIST/test classe (copie)/main.py", line 49, in <module>
    model.add(Dense(18,300))
  File "build/bdist.linux-x86_64/egg/keras/layers/containers.py", line 37, in add
    self.layers[-1].set_previous(self.layers[-2])
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 32, in set_previous
    assert self.input_ndim == len(layer.output_shape), "Incompatible shapes: layer expected input with ndim=" +\
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 489, in output_shape
    return (input_shape[0], np.prod(input_shape[1:]))
  File "/usr/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 2339, in prod
    out=out, keepdims=keepdims)
  File "/usr/lib/python2.7/dist-packages/numpy/core/_methods.py", line 29, in _prod
    out=out, keepdims=keepdims)
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

so I think maybe it's about this line :

model.add(Dense(18,300))

but I don't understand why I couldn't put the dims I want on this layer ...

[dbonadiman]

no with the former i was refering with the input_lenght=window instead of input_shape in to the new code :)

[darzok]

aah lol ! ok so I try this : (I had input dim)

model.add(Convolution1D(nb_filter=nb_filter,
                        filter_length=filter_length,
                        border_mode="valid",
                        activation="relu",
                        input_dim=window,
                        input_length=(window, 1)))
model.add(MaxPooling1D(pool_length=2))
model.add(Dropout(0.25))
}

then I had this :

Traceback (most recent call last):
  File "/home/benjamin/Bureau/test_MNIST/test classe (copie)/main.py", line 44, in <module>
    model.add(MaxPooling1D(pool_length=2))
  File "build/bdist.linux-x86_64/egg/keras/layers/containers.py", line 37, in add
    self.layers[-1].set_previous(self.layers[-2])
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 32, in set_previous
    assert self.input_ndim == len(layer.output_shape), "Incompatible shapes: layer expected input with ndim=" +\
  File "build/bdist.linux-x86_64/egg/keras/layers/convolutional.py", line 114, in output_shape
    length = conv_output_length(self.input_shape[1], self.filter_length, self.border_mode, self.subsample[0])
  File "build/bdist.linux-x86_64/egg/keras/layers/convolutional.py", line 25, in conv_output_length
    output_length = input_length - filter_size + 1
TypeError: unsupported operand type(s) for -: 'tuple' and 'int'
}

so also I tryed this :

model.add(Convolution1D(nb_filter=nb_filter,
                        filter_length=filter_length,
                        border_mode="valid",
                        activation="relu",
                        input_length=(window, 1)))
}

but I had this error :

Traceback (most recent call last):
  File "/home/benjamin/Bureau/test_MNIST/test classe (copie)/main.py", line 43, in <module>
    model.add(MaxPooling1D(pool_length=2))
  File "build/bdist.linux-x86_64/egg/keras/layers/containers.py", line 37, in add
    self.layers[-1].set_previous(self.layers[-2])
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 32, in set_previous
    assert self.input_ndim == len(layer.output_shape), "Incompatible shapes: layer expected input with ndim=" +\
  File "build/bdist.linux-x86_64/egg/keras/layers/convolutional.py", line 114, in output_shape
    length = conv_output_length(self.input_shape[1], self.filter_length, self.border_mode, self.subsample[0])
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 65, in input_shape
    raise Exception('Layer is not connected. Did you forget to set "input_shape"?')
Exception: Layer is not connected. Did you forget to set "input_shape"?
}

.... I don't know what to do ..

[lemuriandezapada]

There's some sort of issue with the Flatten() layer, as it doesn't actually flatten for me (why? the imdb example runs just fine), and retains a (None,None,256) shape for my next layer. Then Dense() on top doesn't work. My skills and understanding of the framework aren't good enough to fix it, so I quick hacked it with using a reshape() instead of flatten() to kill the extra dimensions like:

graph.add_node(MaxPooling1D(pool_length=2), name='mp2', input='conv2')
graph.add_node(Reshape( (14*rsize,) ), name='flat', input='mp2')
graph.add_node(Dense(rsize*4, activation='relu'), name='l0', input='flat')

had to get a few errors to actually figure out what the pooling value would be but now it compiles and trains just fine.

[darzok]

lemuriandezapada could you send your whole code please ? just to understand what did you do and how does it work so maybe I will allow to adapt it to my code.

[lemuriandezapada]

This is the whole structure code. The rest is uninteresting dataloading

graph = Graph()
graph.add_input(name='input', input_shape=[1], dtype='int')
graph.add_node(Embedding(input_dim=vsize, output_dim=embsize), name = 'emb', input = 'input')
graph.add_node(Dropout(drate),name = 'd_emb', input = 'emb')
graph.add_node(Convolution1D(nb_filter=rsize, filter_length=3, border_mode="full", activation="relu", subsample_length=1), name='conv0', input='d_emb')
graph.add_node(MaxPooling1D(pool_length=2), name='mp0', input='conv0')
graph.add_node(Convolution1D(nb_filter=rsize, filter_length=3, border_mode="full", activation="relu", subsample_length=1), name='conv1', input='mp0')
graph.add_node(MaxPooling1D(pool_length=2), name='mp1', input='conv1')
graph.add_node(Convolution1D(nb_filter=rsize, filter_length=3, border_mode="full", activation="relu", subsample_length=1), name='conv2', input='mp1')
graph.add_node(MaxPooling1D(pool_length=2), name='mp2', input='conv2')
graph.add_node(Reshape( (14*rsize,) ), name='flat', input='mp2')
graph.add_node(Dense(rsize*4, activation='relu'), name='l0', input='flat')

graph.add_node(Dropout(drate),name = 'd_l0', input = 'l0')

graph.add_node(Dense(len(classet), activation= 'softmax'), name='finalclass', input='d_l0')
graph.add_output(name='outputclass', input='finalclass')

graph.add_node(Dense(len(groupset), activation= 'softmax'), name='finalgroup', input='d_l0')
graph.add_output(name='outputgroup', input='finalgroup')

graph.add_node(Dense(len(codeset), activation= 'softmax'), name='finalcode', input='d_l0')
graph.add_output(name='outputcode', input='finalcode')

opt = RMSprop(.01)
graph.compile(loss={'outputclass': 'categorical_crossentropy', 'outputgroup': 'categorical_crossentropy', 'outputcode': 'categorical_crossentropy'}, optimizer=opt)

[darzok]

ok I will try but is it normal you put input_shape=[1], that mean you give juste one data at each time right ? for exemple, you data set it's like [15][50]....[45]

[darzok]

.... it doesn't work I completly understand nothing why it doesn't work it just a simple code it has to work !!

my code :

sizeTraining=0.9
sizeTest=0.1
window=59
h=10

rsize = 6

nb_filter = 10
filter_length = 6

print('Loading data ...')
forex = FOREX()
(X_train, y_train),(X_test,y_test) = forex.load_data(sizeTraining, sizeTest, window, h)

X_train = X_train*10000
y_train = y_train*10000

X_test = X_test*10000
y_test = y_test*10000

print(X_train.shape[0],'train samples')
print(X_test.shape[0],'test samples')

graph = Graph()
graph.add_input(name='input', input_shape=[window], dtype='float')

graph.add_node(Convolution1D(nb_filter=rsize, filter_length=filter_length, border_mode="full", activation="relu", subsample_length=1), name='conv0', input='input')
graph.add_node(MaxPooling1D(pool_length=2), name='mp0', input='conv0')
graph.add_node(Reshape( (14*rsize,) ), name='flat', input='mp0')
graph.add_node(Dense(rsize*4, activation='relu'), name='l0', input='flat')
graph.add_node(Dropout(0.2),name = 'd_l0', input = 'l0')
graph.add_node(Dense(rsize*4, activation= 'linear'), name='final', input='d_l0')
graph.add_output(name='outputclass', input='final')

#opt = RMSprop(.01)
graph.compile('rmsprop', {'output':'mse'})

Traceback (most recent call last):
  File "/home/benjamin/Bureau/test_MNIST/test classe (copie)/main.py", line 41, in <module>
    graph.add_node(Convolution1D(nb_filter=rsize, filter_length=filter_length, border_mode="full", activation="relu", subsample_length=1), name='conv0', input='input')
  File "build/bdist.linux-x86_64/egg/keras/layers/containers.py", line 202, in add_node
    layer.set_previous(self.inputs[input])
  File "build/bdist.linux-x86_64/egg/keras/layers/core.py", line 33, in set_previous
    str(self.input_ndim) + " but previous layer has output_shape " + str(layer.output_shape)
AssertionError: Incompatible shapes: layer expected input with ndim=3 but previous layer has output_shape (None, 59)

[lemuriandezapada]

yeah mine is not a time series thing per se. I apply the 1d conv to 100 letter length sentences. I don't think input shape can be 59 dimensional. Max is like 3 for time data (2 if it goes into embedding layer), 2 for no time data (1 if it goes into embedding layer). This isn't the dimensionality of your inputs but more like the shape. For your case I guess dimensionality should be like 2.

[futurely]

The examples apply the Embedding layer to map the input into 2 dim data required by the Convolution1D layer.

print("Pad sequences (samples x time)")
X_train = sequence.pad_sequences(X_train, maxlen=maxlen)
X_test = sequence.pad_sequences(X_test, maxlen=maxlen)
print('X_train shape:', X_train.shape)
print('X_test shape:', X_test.shape)

print('Build model...')
model = Sequential()

# we start off with an efficient embedding layer which maps
# our vocab indices into embedding_dims dimensions
model.add(Embedding(max_features, embedding_dims, input_length=maxlen))
model.add(Dropout(0.25))
``

[futurely]

Convolution1D requires input_shape or both input_dim and input_length.

    model.add(Convolution1D(nb_filter = nb_filter,
                            filter_length = filter_length,
                            border_mode = 'valid',
                            activation = 'relu',
                            subsample_length = 1,
                            # input_dim = input_dim,
                            # input_length = max_len
                            input_shape = (max_len, input_dim)
                            ))