help one regression per sequence

[volvox292]

Hi,

I tried to read through your very nice package and created a new model for training that would fit for my sequences. My x data are sequences of variable length padded to max len of 2002 and the y is my encoded array (512). Were one sequence corresponds/describes one such array of 512. Probably I understand something wrong, I tried different reshaping but could not get the model to predict different arrays. All are the same, currently I train with shape of x = (68015,2002,1), y=(68015,1,512), when I try to predict with x_val (706,1,512), all of the 706 arrays are exactly the same, but each of them should correspond to a different array. I put my current code below. I was also wondering if an attention layer would be useful after the tcn?

Thanks for your help!

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, RepeatVector
import tensorflow as tf
import numpy as np
from tcn import TCN
import wandb
from wandb.keras import WandbCallback

wandb.init(project="tcn_seq")

batch_size=32
n_epochs=10

ytr=np.load('y_train.npy')
ytr=np.expand_dims(ytr,-2)
yval=np.load('y_validation.npy')
yval=np.expand_dims(yval,-2)
xtr=np.load('train_x.npy')

xval=np.load('val_x.npy')

tcn = TCN(
            nb_filters=256,
            kernel_size=3,
            dilations=[2 ** i for i in range(9)],
            use_skip_connections=True,
            use_layer_norm=True,
            kernel_initializer='glorot_uniform'
        )
print(f'TCN.receptive_field: {tcn.receptive_field}.')
model = Sequential(layers=[tf.keras.layers.Masking(mask_value=10,input_shape=(2002,1)),                                       
            tcn,
            RepeatVector(1),               
            Dropout(rate=0.2),
            Dense(512, activation='relu'),
            Dropout(rate=0.2),
            Dense(512, activation='relu'),
            Dropout(rate=0.2),               
            Dense(512, activation='linear')
        ])

    # Compile and train.

model.compile(loss='mean_absolute_error', optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),metrics=['mean_squared_error'])
model.summary()
model.fit(x=xtr,y=ytr,validation_data=(xval,yval), epochs=n_epochs,batch_size=batch_size,shuffle=False,callbacks=[WandbCallback(log_batch_frequency=1)])
result= model.predict(xval)
np.save("val_predict.npy", result)

model.save('train/tcn')

print('done!')

[volvox292]

go_backwards=True solved the problem, probably a problem with masking, as most of the sequences have been padded at the end!