Low accuracy in keras-tuner based TCN model for audio classification

[gehaniaarti]

I am trying to classify audio signals using TCN. For obtaining the TCN hyperparameters, I am using keras-tuner. It runs perfectly fine but I am getting low training and testing accuracies (something like 26% and 17%, respectively). The code is as follows:

num_classes = Y.shape[1]
batch_size = X_train.shape[0]
num_rows = X_train.shape[1]
num_columns = X_train.shape[2]
time_stamp= num_rows * num_columns
input_dim = 1 

X_train = X_train.reshape(X_train.shape[0], time_stamp, input_dim)
X_test = X_test.reshape(X_test.shape[0], time_stamp, input_dim)

def build_model(hp):

      i = Input(batch_shape=(None, time_stamp, input_dim))
      o = TCN(hp.Int('nb_filters', min_value=32, max_value=512, step=32), 
              kernel_size=5, nb_stacks=1, 
              dilations=[1,2,4], return_sequences=False, padding = 'causal', use_skip_connections=True)(i)  # The TCN layers are here.
      o = Dense(hp.Choice('nodes',values=[32,64,128,256]), activation='relu') (o)
      o = Dropout(hp.Choice('dropout',values=[0.1,0.2]))(o)
      o = Dense(num_classes, activation = 'softmax')(o)

      model = Model(inputs=[i], outputs=[o])

      model.summary()

      model.compile(loss='categorical_crossentropy', 
                    metrics=['accuracy'], 
                    optimizer='adam')

      return model

tuner = RandomSearch(build_model, objective='val_accuracy',
                    max_trials = 3, executions_per_trial=1, overwrite=True)
tuner.search_space_summary()

tuner.search(x=X_train, y=Y_train, epochs=50, batch_size=32, validation_data=(X_test, Y_test))
tuner.results_summary()  

best_hps=tuner.get_best_hyperparameters(num_trials=1)[0]
print(best_hps.values)  

model = tuner.hypermodel.build(best_hps)
history = model.fit(X_train, Y_train, epochs=50, validation_split=0.2)

val_acc_per_epoch = history.history['val_accuracy']
best_epoch = val_acc_per_epoch.index(max(val_acc_per_epoch)) + 1
print('Best epoch: %d' % (best_epoch,))

hypermodel = tuner.hypermodel.build(best_hps)

hypermodel.fit(X_train, Y_train, epochs=best_epoch)

eval_result = hypermodel.evaluate(X_test, Y_test)
print("[test loss, test accuracy]:", eval_result)

The shapes for the training, testing and validation datasets are as follows: shape of X_train is: (219, 99, 32), shape of X_Val is: (27, 99, 32), shape of X_Test is: (28, 99, 32), shape of Y_train is: (219, 5), shape of Y_Val is: (27, 5), shape of Y_Test is: (28, 5)

A similar approach gave me good results with CNN, CRNN and MLP but is not working here.

[philipperemy]

@gehaniaarti have you tried with the default parameters? It looks like your dataset is very small. TCN are harder to train than CNN and MLP. If you look at the examples, TCN need tens of thousands of example to perform well. Also what data are you using for audio classification? Spectrogram? Wavelengths?

[gehaniaarti]

What do you mean by default parameters?

Yes, the dataset is smaller as that is our targeted scenario. Does that mean, I won’t get good accuracy using TCN on smaller datasets?

On Fri, Aug 4, 2023 at 7:21 PM Philippe Rémy @.***> wrote:

@gehaniaarti https://github.com/gehaniaarti have you tried with the default parameters? It looks like your dataset is very small. TCN are harder to train than CNN and MLP. If you look at the examples, TCN need tens of thousands of example to perform well.

— Reply to this email directly, view it on GitHub https://github.com/philipperemy/keras-tcn/issues/248#issuecomment-1666354585, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANSBUCO5MK4HEVUM6U6TFNLXTWUZFANCNFSM6AAAAAA235S72M . You are receiving this because you were mentioned.Message ID: @.***>

[philipperemy]

Default parameters means just calling TCN() like this. Yes I think your dataset is too small to have something that can work reasonably well.

[gehaniaarti]

Ok, let me give it a try and get back to you with updates.

Thanks, Aarti

On Sun, Aug 6, 2023 at 9:34 PM Philippe Rémy @.***> wrote:

Default parameters means just calling TCN() like this. Yes I think your dataset is too small to have something that can work reasonably well.

— Reply to this email directly, view it on GitHub https://github.com/philipperemy/keras-tcn/issues/248#issuecomment-1667170304, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANSBUCICCLQW2KE4YM4JNZTXUBV6PANCNFSM6AAAAAA235S72M . You are receiving this because you were mentioned.Message ID: @.***>

[gehaniaarti]

I tried using the default parameters but still it gives me low accuracy (35% training accuracy and 21% testing accuracy). Although it is better than when I was not using the default parameters but still is not good.

Can we conclude that since the dataset is limited we are getting low accuracy? Or is there anything that I am missing?

On Mon, Aug 7, 2023 at 1:46 PM Aarti Gehani @.***> wrote:

Ok, let me give it a try and get back to you with updates.

Thanks, Aarti

On Sun, Aug 6, 2023 at 9:34 PM Philippe Rémy @.***> wrote:

Default parameters means just calling TCN() like this. Yes I think your dataset is too small to have something that can work reasonably well.

— Reply to this email directly, view it on GitHub https://github.com/philipperemy/keras-tcn/issues/248#issuecomment-1667170304, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANSBUCICCLQW2KE4YM4JNZTXUBV6PANCNFSM6AAAAAA235S72M . You are receiving this because you were mentioned.Message ID: @.***>

[philipperemy]

Yes the dataset is too small.