2 second window, changelog

ambitious-octopus / MI-EEG-1D-CNN

A new approach based on a 10-layer one-dimensional convolution neural network (1D-CNN) to classify five brain states (four MI classes plus a 'baseline' class) using a data augmentation algorithm and a limited number of EEG channels. Paper: https://doi.org/10.1088/1741-2552/ac4430

GNU General Public License v3.0

158 stars 18 forks source link

2 second window, changelog #15

Closed ambitious-octopus closed 2 years ago

ambitious-octopus commented 3 years ago

channgelog -> 08/02 learning_rate = 1e-4 -> 1e-3 Added callbacks, Early Stopping and Checkpoints epochs = 150 -> 200

ambitious-octopus commented 3 years ago

SMOTE -> 89% NO-SMOTE -> 89%

stefano-bargione commented 3 years ago

First changes to the neural net (data loaded in the net through the folder nosmotetest1):

-second Conv1D layer(from the top of architecture structure) from "padding = same" -> to "padding = valid" -40 epochs of training -model saved in file "bestModel1.h5" -training history at -> test1.out

Results: The training process goes in overfitting as it could be detected from the trend

stefano-bargione commented 3 years ago

Second changes to the neural net (data loaded in the net through the folder nosmotetest1):

-first Conv1D layer(from the top of architecture structure) from "padding = same" -> to "padding = valid" -40 epochs of training -model saved in file "bestModel2.h5" -training history at -> test2.out

Results: The training process goes in overfitting as it could be detected from the trend

stefano-bargione commented 3 years ago

Third changes to the neural net (data loaded in the net through the folder nosmotetest1):

-first Conv1D layer(from the top of architecture structure) from "padding = valid" -> back to "padding = same"

-kernel_size_0 from 4 to 6 -kernel_size_1 = from 2 to 3

-40 epochs of training -model saved in file "bestModel3.h5" -training history at -> test3.out

Results: The training process goes in overfitting as it could be detected from the trend

stefano-bargione commented 3 years ago

Fourth changes to the neural net (data loaded in the net through the folder nosmotetest1):

-kernel_size_0 from 6 to 4 (so back to default parameter value) -kernel_size_1 = from 3 to 2 (so back to default parameter value)

-second Conv1D layer(from the top of architecture structure) from parameter 'filter' = 32-> 'filter' = 28 -third Conv1D layer(from the top of architecture structure) from parameter 'filter' = 32-> 'filter' = 28 -fourth Conv1D layer(from the top of architecture structure) from parameter 'filter' = 32-> 'filter' = 28

-40 epochs of training -model saved in file "bestModel4.h5" -training history at -> test4.out

Results: The training process goes in overfitting as it could be detected from the trend

stefano-bargione commented 3 years ago

Fifth changes to the neural net (data loaded in the net through the folder nosmotetest1):

-second Conv1D layer(from the top of architecture structure) from parameter 'filter' = 28-> 'filter' = 30 -third Conv1D layer(from the top of architecture structure) from parameter 'filter' = 28-> 'filter' = 30 -fourth Conv1D layer(from the top of architecture structure) from parameter 'filter' = 28-> 'filter' = 30

-40 epochs of training -model saved in file "bestModel5.h5" -training history at -> test5.out


learning_rate = 1e-4 # default 1e-3

loss = tf.keras.losses.categorical_crossentropy  #tf.keras.losses.categorical_crossentropy
optimizer = tf.keras.optimizers.Adam(lr=learning_rate)

kernel_size_0 = 4
kernel_size_1 = 2
drop_rate = 0.5

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv1D(filters=32,
                                kernel_size=kernel_size_0,
                                activation='relu',
                                padding= "same",
                                input_shape=(320, 2)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv1D(filters=30,
                                            kernel_size=kernel_size_0,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Conv1D(filters=30,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.AvgPool1D(pool_size=2))
model.add(tf.keras.layers.Conv1D(filters=30,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(296, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(148, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(74, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(5, activation='softmax'))

model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

model.summary()

Results: The training process goes in overfitting as it could be detected from the trend

ambitious-octopus commented 3 years ago

I will leave it run for some more epoch!

learning_rate = 1e-4 # default 1e-3

loss = tf.keras.losses.categorical_crossentropy  #tf.keras.losses.categorical_crossentropy
optimizer = tf.keras.optimizers.Adam(lr=learning_rate)

kernel_size_0 = 4
kernel_size_1 = 2
drop_rate = 0.5

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv1D(filters=32,
                                kernel_size=kernel_size_0,
                                activation='relu',
                                padding= "same",
                                input_shape=(320, 2)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_0,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.AvgPool1D(pool_size=2))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(296, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(148, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(74, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(5, activation='softmax'))

model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

model.summary()

Epoch 29/200
5968/5968 [==============================] - ETA: 0s - loss: 1.4451 - accuracy: 0.3721
Epoch 00029: val_loss improved from 1.40232 to 1.38737, saving model to C:\Users\franc_pyl533c\OneDrive\Repository\eeGNN\bestModel.h5
5968/5968 [==============================] - 103s 17ms/step - loss: 1.4451 - accuracy: 0.3721 - val_loss: 1.3874 - val_accuracy: 0.4388
Epoch 30/200
5966/5968 [============================>.] - ETA: 0s - loss: 1.4327 - accuracy: 0.3781
Epoch 00030: val_loss improved from 1.38737 to 1.37226, saving model to C:\Users\franc_pyl533c\OneDrive\Repository\eeGNN\bestModel.h5
5968/5968 [==============================] - 108s 18ms/step - loss: 1.4327 - accuracy: 0.3781 - val_loss: 1.3723 - val_accuracy: 0.4532
Epoch 31/200
5965/5968 [============================>.] - ETA: 0s - loss: 1.4210 - accuracy: 0.3844
Epoch 00031: val_loss improved from 1.37226 to 1.35286, saving model to C:\Users\franc_pyl533c\OneDrive\Repository\eeGNN\bestModel.h5
5968/5968 [==============================] - 100s 17ms/step - loss: 1.4210 - accuracy: 0.3844 - val_loss: 1.3529 - val_accuracy: 0.4659
Epoch 32/200
5968/5968 [==============================] - ETA: 0s - loss: 1.4056 - accuracy: 0.3924
Epoch 00032: val_loss improved from 1.35286 to 1.34032, saving model to C:\Users\franc_pyl533c\OneDrive\Repository\eeGNN\bestModel.h5
5968/5968 [==============================] - 100s 17ms/step - loss: 1.4056 - accuracy: 0.3924 - val_loss: 1.3403 - val_accuracy: 0.4737
Epoch 33/200
5967/5968 [============================>.] - ETA: 0s - loss: 1.3919 - accuracy: 0.4005
Epoch 00033: val_loss improved from 1.34032 to 1.31838, saving model to C:\Users\franc_pyl533c\OneDrive\Repository\eeGNN\bestModel.h5
5968/5968 [==============================] - 103s 17ms/step - loss: 1.3919 - accuracy: 0.4005 - val_loss: 1.3184 - val_accuracy: 0.4855
Epoch 34/200
5966/5968 [============================>.] - ETA: 0s - loss: 1.3797 - accuracy: 0.4065
Epoch 00034: val_loss improved from 1.31838 to 1.30800, saving model to C:\Users\franc_pyl533c\OneDrive\Repository\eeGNN\bestModel.h5
5968/5968 [==============================] - 107s 18ms/step - loss: 1.3797 - accuracy: 0.4065 - val_loss: 1.3080 - val_accuracy: 0.4964
Epoch 35/200
5966/5968 [============================>.] - ETA: 0s - loss: 1.3671 - accuracy: 0.4147
Epoch 00035: val_loss improved from 1.30800 to 1.29152, saving model to C:\Users\franc_pyl533c\OneDrive\Repository\eeGNN\bestModel.h5
5968/5968 [==============================] - 102s 17ms/step - loss: 1.3671 - accuracy: 0.4147 - val_loss: 1.2915 - val_accuracy: 0.5050
Epoch 36/200
5967/5968 [============================>.] - ETA: 0s - loss: 1.3530 - accuracy: 0.4206
Epoch 00036: val_loss improved from 1.29152 to 1.26736, saving model to C:\Users\franc_pyl533c\OneDrive\Repository\eeGNN\bestModel.h5
5968/5968 [==============================] - 96s 16ms/step - loss: 1.3530 - accuracy: 0.4206 - val_loss: 1.2674 - val_accuracy: 0.5090
Epoch 37/200
5967/5968 [============================>.] - ETA: 0s - loss: 1.3447 - accuracy: 0.4251
Epoch 00037: val_loss improved from 1.26736 to 1.25959, saving model to C:\Users\franc_pyl533c\OneDrive\Repository\eeGNN\bestModel.h5
5968/5968 [==============================] - 98s 16ms/step - loss: 1.3447 - accuracy: 0.4251 - val_loss: 1.2596 - val_accuracy: 0.5190
Epoch 38/200
5968/5968 [==============================] - ETA: 0s - loss: 1.3329 - accuracy: 0.4312
Epoch 00038: val_loss improved from 1.25959 to 1.25693, saving model to C:\Users\franc_pyl533c\OneDrive\Repository\eeGNN\bestModel.h5
5968/5968 [==============================] - 98s 16ms/step - loss: 1.3329 - accuracy: 0.4312 - val_loss: 1.2569 - val_accuracy: 0.5230
Epoch 39/200
5968/5968 [==============================] - ETA: 0s - loss: 1.3154 - accuracy: 0.4425
Epoch 00039: val_loss improved from 1.25693 to 1.23332, saving model to C:\Users\franc_pyl533c\OneDrive\Repository\eeGNN\bestModel.h5
5968/5968 [==============================] - 97s 16ms/step - loss: 1.3154 - accuracy: 0.4425 - val_loss: 1.2333 - val_accuracy: 0.5334

stefano-bargione commented 3 years ago

Fifth changes to the neural net (data loaded in the net through the folder nosmotetest1):

back to original HopefullNet default structure therefore:

-kernel_size_0 = 20 -kernel_size_1 = 6 -padding of the second, third and fourth Conv1D layer (from the top of architecture stucture) is set to previous default values, so from parameter 'padding' = same -> 'padding' = valid

subsequently, additional changes from the previous tested architecture to back off to the default HopefullNet architecture:

-second Conv1D layer(from the top of architecture structure) from parameter 'filter' = 28-> 'filter' = 32 -third Conv1D layer(from the top of architecture structure) from parameter 'filter' = 28-> 'filter' = 32 -fourth Conv1D layer(from the top of architecture structure) from parameter 'filter' = 28-> 'filter' = 32

BUT since we take 2 seconds window, further refinements are drawn from the default HopefullNet, as it displayed below:

-added a parameter 'seed = 2' to set default initialization of kernel -changed the parameter 'kernel_size_0 = 16' instead of 'kernel_size = 20' -set unaltered the parameter 'kernel_size_1 = 6'

-for any Conv1D layer of the architecture, we set a parameter 'kernel_initializer= tf.keras.initializers.HeUniform(seed = seed))'

-40 epochs of training -model saved in file "bestModel6.h5" -training history at -> test6.out


loss = tf.keras.losses.categorical_crossentropy  #tf.keras.losses.categorical_crossentropy
optimizer = tf.keras.optimizers.Adam(lr=learning_rate)

kernel_size_0 = 16
kernel_size_1 = 6
drop_rate = 0.5
seed = 2

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv1D(kernel_initializer= tf.keras.initializers.HeUniform(seed = seed)),
                                filters=32,
                                kernel_size=kernel_size_0,
                                activation='relu',
                                padding= "same",
                                input_shape=(320, 2)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv1D(kernel_initializer = tf.keras.initializers.HeUniform(seed = seed),
                                            filters=32,
                                            kernel_size=kernel_size_0,
                                            activation='relu',
                                            padding= "valid"))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Conv1D(kernel_initializer = tf.keras.initializers.HeUniform(seed = seed),
                                            filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "valid"))
model.add(tf.keras.layers.AvgPool1D(pool_size=2))
model.add(tf.keras.layers.Conv1D(kernel_initializer= tf.keras.initializers.HeUniform(seed = seed),
                                            filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "valid"))
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(296, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(148, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(74, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(5, activation='softmax'))

model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

model.summary()

Results: The training process goes in overfitting as it could be detected from the trend

stefano-bargione commented 3 years ago

Based on Kubasinska suggestions,

the net shown in the "GlobalAveragePooling1D seems to work" comment is experimented with few changes:

-kernel_size_0 = 16 -kernel_size_1 = 6

-GlobalAveragePooling1D layer after second Conv1D layer(from the top of architecture structure)

-padding of the second, third and fourth Conv1D layer (from the top of architecture stucture) is set to previous default values of the HopefullNet, so from parameter 'padding' = same -> 'padding' = valid

-55 epochs of training rather than 40 -model saved in file "bestModel7.h5" -training history at -> test7.out -batch size put to 38 for both train and test set (rather than 32!)


#%%
learning_rate = 1e-4 # default 1e-3

loss = tf.keras.losses.categorical_crossentropy  #tf.keras.losses.categorical_crossentropy
optimizer = tf.keras.optimizers.Adam(lr=learning_rate)

kernel_size_0 = 16
kernel_size_1 = 6
drop_rate = 0.5

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv1D(filters=32,
                                kernel_size=kernel_size_0,
                                activation='relu',
                                padding= "same",
                                input_shape=(320, 2)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_0,
                                            activation='relu',
                                            padding= "valid"))
tf.keras.layers.GlobalAveragePooling1D(data_format='channels_last')
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "valid"))
model.add(tf.keras.layers.AvgPool1D(pool_size=2))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "valid"))
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(296, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(148, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(74, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(5, activation='softmax'))

model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

model.summary()

modelPath = os.path.join(os.getcwd(),'bestModel7.h5')

checkpoint = ModelCheckpoint( # set model saving checkpoints
    modelPath, # set path to save model weights
    monitor='val_loss', # set monitor metrics
    verbose=1, # set training verbosity
    save_best_only=True, # set if want to save only best weights
    save_weights_only=False, # set if you want to save only model weights
    mode='auto', # set if save min or max in metrics
    period=1 # interval between checkpoints
    )

earlystopping = EarlyStopping(
    monitor='val_loss', # set monitor metrics
    min_delta = 0.001, # set minimum metrics delta
    patience = 15, # number of epochs to stop training
    restore_best_weights = True, # set if use best weights or last weights
    )
callbacksList = [checkpoint, earlystopping] # build callbacks list
#%

hist = model.fit(x = x_train.batch(38), epochs = 55, validation_data = x_test.batch(38), callbacks = callbacksList) #32

Results: The training process goes in overfitting as it could be detected from the trend

stefano-bargione commented 3 years ago

Based on Kubasinska suggestions,

the net shown in the "GlobalAveragePooling1D seems to work" comment is experimented with few changes:

-kernel_size_0 = 4 -kernel_size_1 = 2

-GlobalAveragePooling1D layer after second Conv1D layer(from the top of architecture structure)

-padding of the second, third and fourth Conv1D layer (from the top of architecture stucture) change from parameter 'padding' = valid -> 'padding' = same

-45 epochs of training rather than 55 -model saved in file "bestModel8.h5" -training history at -> test8.out -batch size put to 32 for both train and test set (rather than 32!)


learning_rate = 1e-4 # default 1e-3

loss = tf.keras.losses.categorical_crossentropy  #tf.keras.losses.categorical_crossentropy
optimizer = tf.keras.optimizers.Adam(lr=learning_rate)

kernel_size_0 = 4
kernel_size_1 = 2
drop_rate = 0.5

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv1D(filters=32,
                                kernel_size=kernel_size_0,
                                activation='relu',
                                padding= "same",
                                input_shape=(320, 2)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_0,
                                            activation='relu',
                                            padding= "same"))
tf.keras.layers.GlobalAveragePooling1D(data_format='channels_last')
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.AvgPool1D(pool_size=2))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(296, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(148, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(74, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(5, activation='softmax'))

model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

model.summary()

modelPath = os.path.join(os.getcwd(),'bestModel8.h5')

checkpoint = ModelCheckpoint( # set model saving checkpoints
    modelPath, # set path to save model weights
    monitor='val_loss', # set monitor metrics
    verbose=1, # set training verbosity
    save_best_only=True, # set if want to save only best weights
    save_weights_only=False, # set if you want to save only model weights
    mode='auto', # set if save min or max in metrics
    period=1 # interval between checkpoints
    )

earlystopping = EarlyStopping(
    monitor='val_loss', # set monitor metrics
    min_delta = 0.001, # set minimum metrics delta
    patience = 15, # number of epochs to stop training
    restore_best_weights = True, # set if use best weights or last weights
    )
callbacksList = [checkpoint, earlystopping] # build callbacks list
#%

hist = model.fit(x = x_train.batch(32), epochs = 45, validation_data = x_test.batch(32), callbacks = callbacksList) #32

Results: The training process goes in overfitting as it could be detected from the trend

stefano-bargione commented 3 years ago

9th attempt for the net, with respect to the previous tested net some changes are provided:

-kernel_size_0 = 18 -kernel_size_1 = 6

-45 epochs of training -model saved in file "bestModel9.h5" -training history at -> test9.out


learning_rate = 1e-4 # default 1e-3

loss = tf.keras.losses.categorical_crossentropy  #tf.keras.losses.categorical_crossentropy
optimizer = tf.keras.optimizers.Adam(lr=learning_rate)

kernel_size_0 = 18
kernel_size_1 = 6
drop_rate = 0.5

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv1D(filters=32,
                                kernel_size=kernel_size_0,
                                activation='relu',
                                padding= "same",
                                input_shape=(320, 2)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_0,
                                            activation='relu',
                                            padding= "same"))
tf.keras.layers.GlobalAveragePooling1D(data_format='channels_last')
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.AvgPool1D(pool_size=2))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(296, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(148, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(74, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(5, activation='softmax'))

model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

model.summary()

modelPath = os.path.join(os.getcwd(),'bestModel9.h5')

checkpoint = ModelCheckpoint( # set model saving checkpoints
    modelPath, # set path to save model weights
    monitor='val_loss', # set monitor metrics
    verbose=1, # set training verbosity
    save_best_only=True, # set if want to save only best weights
    save_weights_only=False, # set if you want to save only model weights
    mode='auto', # set if save min or max in metrics
    period=1 # interval between checkpoints
    )

earlystopping = EarlyStopping(
    monitor='val_loss', # set monitor metrics
    min_delta = 0.001, # set minimum metrics delta
    patience = 15, # number of epochs to stop training
    restore_best_weights = True, # set if use best weights or last weights
    )
callbacksList = [checkpoint, earlystopping] # build callbacks list
#%

hist = model.fit(x = x_train.batch(32), epochs = 45, validation_data = x_test.batch(32), callbacks = callbacksList) #32

Results: The training process goes in overfitting as it could be detected from the trend

stefano-bargione commented 3 years ago

10th attempt for the net, changes are returned back to the default HopefullNet architecture with slight changes:

-kernel_size_0 = 20 -kernel_size_1 = 6

-GlobalAveragePooling1D layer after second Conv1D layer(from the top of architecture structure)

-padding of the second, third and fourth Conv1D layer (from the top of architecture stucture) change from parameter 'padding' = same -> 'padding' = valid as the original Hopefullnet default values

-batch size for train and test 30 other than 32

-45 epochs of training -model saved in file "bestModel10.h5" -training history at -> test10.out


learning_rate = 1e-4 # default 1e-3

loss = tf.keras.losses.categorical_crossentropy  #tf.keras.losses.categorical_crossentropy
optimizer = tf.keras.optimizers.Adam(lr=learning_rate)

kernel_size_0 = 20
kernel_size_1 = 6
drop_rate = 0.5

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv1D(filters=32,
                                kernel_size=kernel_size_0,
                                activation='relu',
                                padding= "same",
                                input_shape=(320, 2)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_0,
                                            activation='relu',
                                            padding= "valid"))
tf.keras.layers.GlobalAveragePooling1D(data_format='channels_last')
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "valid"))
model.add(tf.keras.layers.AvgPool1D(pool_size=2))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "valid"))
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(296, activation='relu')) 
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(148, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(74, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(5, activation='softmax'))

model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

model.summary()

modelPath = os.path.join(os.getcwd(),'bestModel10.h5')

checkpoint = ModelCheckpoint( # set model saving checkpoints
    modelPath, # set path to save model weights
    monitor='val_loss', # set monitor metrics
    verbose=1, # set training verbosity
    save_best_only=True, # set if want to save only best weights
    save_weights_only=False, # set if you want to save only model weights
    mode='auto', # set if save min or max in metrics
    period=1 # interval between checkpoints
    )
earlystopping = EarlyStopping(
    monitor='val_loss', # set monitor metrics
    min_delta = 0.001, # set minimum metrics delta
    patience = 15, # number of epochs to stop training
    restore_best_weights = True, # set if use best weights or last weights
    )
callbacksList = [checkpoint, earlystopping] # build callbacks list 
#%

hist = model.fit(x = x_train.batch(30), epochs = 45, validation_data = x_test.batch(30), callbacks = callbacksList) #32

Schermata 2021-06-21 alle 15 04 17

stefano-bargione commented 3 years ago

11h attempt for the net, changes are returned back to the default HopefullNet architecture provided by @Kubasinska as shown below, specifically:

-kernel_size_0 = 4 -kernel_size_1 = 2

-GlobalAveragePooling1D layer after second Conv1D layer(from the top of architecture structure)

-padding of the second, third and fourth Conv1D layer (from the top of architecture stucture) change from parameter 'padding' = valid -> 'padding' = same

-batch size for train and test 32 rather than 30

-55 epochs of training -model saved in file "bestModel11.h5" -training history at -> test11.out


learning_rate = 1e-4 # default 1e-3

loss = tf.keras.losses.categorical_crossentropy  #tf.keras.losses.categorical_crossentropy
optimizer = tf.keras.optimizers.Adam(lr=learning_rate)

kernel_size_0 = 4
kernel_size_1 = 2
drop_rate = 0.5

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv1D(filters=32,
                                kernel_size=kernel_size_0,
                                activation='relu',
                                padding= "same",
                                input_shape=(320, 2)))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_0,
                                            activation='relu',
                                            padding= "same"))
tf.keras.layers.GlobalAveragePooling1D(data_format='channels_last')
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.AvgPool1D(pool_size=2))
model.add(tf.keras.layers.Conv1D(filters=32,
                                            kernel_size=kernel_size_1,
                                            activation='relu',
                                            padding= "same"))
model.add(tf.keras.layers.SpatialDropout1D(drop_rate))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense(296, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(148, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(74, activation='relu'))
model.add(tf.keras.layers.Dropout(drop_rate))
model.add(tf.keras.layers.Dense(5, activation='softmax'))

model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])

model.summary()

Results: The training process goes in overfitting as it could be detected from the trend