noise threshold

[geoffwoollard]

I made some synthetic data of a particle +/- about 5% of it missing. You can distinguish it by eye, but not all the time, especially when the present part is overlapping.

I modified the architecture to have a bigger filter, and extra layer, among other things.

params = architecture_params_wrapper(blocks=6,
                                inputl_kernel=30,
                                inputl_node=8,
                                hiddenl_kernels=(15,7,5),
                                hiddenl_nodes=((8,8),(8,16),(32,32)),
                                pooling_sizes = (15,7,5,3),
                                n1_ave_pool=64, 
                                n2_ave_pool=64, 
                                k1_ave_pool=3, 
                                k2_ave_pool=3, 
                                av_pool=4,
                                val_dense=256,
                                drop=0.5
                               )
input_shape=X.shape[1::]
n_classes = 2

# Call function build_network

model = customizable_deep_network.build_network (
  blocks=params['blocks'],
  input_shape=input_shape, 
  inputl_node=params['inputl_node'],
  inputl_kernel=params['inputl_kernel'],
  hiddenl_node=params['hiddenl_node'],

  hiddenl_kernel=params['hiddenl_kernel'], 
  pooling_size=params['pooling_size'], 
  n1_ave_pool=params['n1_ave_pool'], 
  n2_ave_pool=params['n2_ave_pool'], 

  k1_ave_pool=params['k1_ave_pool'], 
  k2_ave_pool=params['k2_ave_pool'], 
  av_pool=params['av_pool'], 
  val_dense=params['val_dense'], 
  drop=params['drop'], 
  n_classes=n_classes)

print(model.summary())
model.compile(optimizer = SGD() , loss='categorical_crossentropy', metrics=['categorical_accuracy'])

Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_19 (Conv2D)           (None, 380, 380, 8)       7208      
_________________________________________________________________
conv2d_20 (Conv2D)           (None, 380, 380, 8)       57608     
_________________________________________________________________
batch_normalization_10 (Batc (None, 380, 380, 8)       32        
_________________________________________________________________
activation_10 (Activation)   (None, 380, 380, 8)       0         
_________________________________________________________________
max_pooling2d_8 (MaxPooling2 (None, 190, 190, 8)       0         
_________________________________________________________________
conv2d_21 (Conv2D)           (None, 190, 190, 8)       14408     
_________________________________________________________________
conv2d_22 (Conv2D)           (None, 190, 190, 8)       14408     
_________________________________________________________________
batch_normalization_11 (Batc (None, 190, 190, 8)       32        
_________________________________________________________________
activation_11 (Activation)   (None, 190, 190, 8)       0         
_________________________________________________________________
max_pooling2d_9 (MaxPooling2 (None, 95, 95, 8)         0         
_________________________________________________________________
conv2d_23 (Conv2D)           (None, 95, 95, 8)         3144      
_________________________________________________________________
conv2d_24 (Conv2D)           (None, 95, 95, 16)        6288      
_________________________________________________________________
batch_normalization_12 (Batc (None, 95, 95, 16)        64        
_________________________________________________________________
activation_12 (Activation)   (None, 95, 95, 16)        0         
_________________________________________________________________
max_pooling2d_10 (MaxPooling (None, 48, 48, 16)        0         
_________________________________________________________________
conv2d_25 (Conv2D)           (None, 48, 48, 32)        12832     
_________________________________________________________________
conv2d_26 (Conv2D)           (None, 48, 48, 32)        25632     
_________________________________________________________________
batch_normalization_13 (Batc (None, 48, 48, 32)        128       
_________________________________________________________________
activation_13 (Activation)   (None, 48, 48, 32)        0         
_________________________________________________________________
max_pooling2d_11 (MaxPooling (None, 24, 24, 32)        0         
_________________________________________________________________
conv2d_27 (Conv2D)           (None, 24, 24, 64)        18496     
_________________________________________________________________
conv2d_28 (Conv2D)           (None, 24, 24, 64)        36928     
_________________________________________________________________
batch_normalization_14 (Batc (None, 24, 24, 64)        256       
_________________________________________________________________
activation_14 (Activation)   (None, 24, 24, 64)        0         
_________________________________________________________________
average_pooling2d_3 (Average (None, 12, 12, 64)        0         
_________________________________________________________________
flatten_3 (Flatten)          (None, 9216)              0         
_________________________________________________________________
dense_5 (Dense)              (None, 256)               2359552   
_________________________________________________________________
dropout_3 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_6 (Dense)              (None, 2)                 514       
=================================================================
Total params: 2,557,530
Trainable params: 2,557,274
Non-trainable params: 256
_________________________________________________________________
None

It only takes 2-3 epochs to train.

epoch 0
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Epoch 1/1
18/18 [==============================] - 444s 25s/step - loss: 0.8363 - categorical_accuracy: 0.7867
200/200 [==============================] - 32s 158ms/step
categorical_accuracy: 73.00%
epoch 1
Epoch 1/1
18/18 [==============================] - 364s 20s/step - loss: 0.0163 - categorical_accuracy: 1.0000
200/200 [==============================] - 13s 67ms/step
categorical_accuracy: 100.00%
epoch 2
Epoch 1/1
18/18 [==============================] - 364s 20s/step - loss: 0.0069 - categorical_accuracy: 1.0000
200/200 [==============================] - 13s 67ms/step
categorical_accuracy: 100.00%
epoch 3
Epoch 1/1
18/18 [==============================] - 364s 20s/step - loss: 0.0046 - categorical_accuracy: 1.0000
200/200 [==============================] - 13s 67ms/step
categorical_accuracy: 100.00%
epoch 4
Epoch 1/1
18/18 [==============================] - 364s 20s/step - loss: 0.0030 - categorical_accuracy: 1.0000
200/200 [==============================] - 13s 67ms/step
categorical_accuracy: 100.00%

I am going to increase the noise threshold and see how far it can be pushed.

[geoffwoollard]

Interestingly, the following network fits to 100%, but trains at 50%. It overfits. It seems the large filter size enables the network to memorize the images in the training set. This was the only thing that changed.

params = architecture_params_wrapper(blocks=6, 
                                inputl_kernel=45,
                                inputl_node=8,
                                hiddenl_kernels=(15,7,5),
                                hiddenl_nodes=((8,8),(8,16),(32,32)),
                                pooling_sizes = (15,7,5,3), 
                                n1_ave_pool=64, 
                                n2_ave_pool=64, 
                                k1_ave_pool=3, 
                                k2_ave_pool=3, 
                                av_pool=4,
                                val_dense=256,
                                drop=0.5 
                               )

[geoffwoollard]

@Davjes15 The train and test sets have not been properly split in probably all the notebooks, dueto a bug.

The initial setup was

df = fit_generator_helper.parse_files(fname_list,label_list)
df = df.shape[0])
header = mrc.read_header(fname_list[0])
nx,ny=header['nx'],header['nx']
val_n = int(0.1*df.shape[0]) # 10%/90% test/train split
df = df.iloc[:-val_n]
df_val = df.iloc[-val_n:]

But if you notice df and df_val share the same data! We should have something like:

df = fit_generator_helper.parse_files(fname_list,label_list)
df = df.sample(df.shape[0])
header = mrc.read_header(fname_list[0])
nx,ny=header['nx'],header['nx']
val_n = int(0.1*df.shape[0]) # 10%/90% test/train split
df_train = df.iloc[:-val_n]
df_val = df.iloc[-val_n:]

With df_train being used

[Davjes15]

@geoffwoollard Ok I am going to fix that and run the model again

[geoffwoollard]

@Davjes15 include in table. I used this architecture for the synthetic data and got 100% accuracy on the validation data.

model = deepconsensus_layers_wrapper(
    input_shape=X_val.shape[1::],
    num_hidden_layers=4,

    conv2d1_k=(30,15,7,3),
    conv2d2_k=(30,15,7,3),
    conv2d1_n=(8,8,32,64),
    conv2d2_n=(8,16,32,64),
    mp_k=(3,3,3,4),
    mp_strides=(4,4,2,2),
  pooling_type=('max','max','max','av'),
)
model.summary()

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 256, 256, 8)       7208      
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 256, 256, 8)       57608     
_________________________________________________________________
batch_normalization_1 (Batch (None, 256, 256, 8)       32        
_________________________________________________________________
activation_1 (Activation)    (None, 256, 256, 8)       0         
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 64, 64, 8)         0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 64, 64, 8)         14408     
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 64, 64, 16)        28816     
_________________________________________________________________
batch_normalization_2 (Batch (None, 64, 64, 16)        64        
_________________________________________________________________
activation_2 (Activation)    (None, 64, 64, 16)        0         
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 16, 16, 16)        0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 16, 16, 32)        25120     
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 16, 16, 32)        50208     
_________________________________________________________________
batch_normalization_3 (Batch (None, 16, 16, 32)        128       
_________________________________________________________________
activation_3 (Activation)    (None, 16, 16, 32)        0         
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 8, 8, 32)          0         
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 8, 8, 64)          18496     
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 8, 8, 64)          36928     
_________________________________________________________________
batch_normalization_4 (Batch (None, 8, 8, 64)          256       
_________________________________________________________________
activation_4 (Activation)    (None, 8, 8, 64)          0         
_________________________________________________________________
average_pooling2d_1 (Average (None, 4, 4, 64)          0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 1024)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 512)               524800    
_________________________________________________________________
dropout_1 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 2)                 1026      
=================================================================
Total params: 765,098
Trainable params: 764,858
Non-trainable params: 240

[geoffwoollard]

I couldn't get above random on validation accuracy for the slightly more noisy data (0.4 vs 0.5 snr) with the same architecture.

However, when I doubled the dense layer from 512 to 1054 I could get above random accuracy, but then it dropped in later epochs.

Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_9 (Conv2D)            (None, 256, 256, 8)       7208      
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 256, 256, 8)       57608     
_________________________________________________________________
batch_normalization_5 (Batch (None, 256, 256, 8)       32        
_________________________________________________________________
activation_5 (Activation)    (None, 256, 256, 8)       0         
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 64, 64, 8)         0         
_________________________________________________________________
conv2d_11 (Conv2D)           (None, 64, 64, 8)         14408     
_________________________________________________________________
conv2d_12 (Conv2D)           (None, 64, 64, 16)        28816     
_________________________________________________________________
batch_normalization_6 (Batch (None, 64, 64, 16)        64        
_________________________________________________________________
activation_6 (Activation)    (None, 64, 64, 16)        0         
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 16, 16, 16)        0         
_________________________________________________________________
conv2d_13 (Conv2D)           (None, 16, 16, 32)        25120     
_________________________________________________________________
conv2d_14 (Conv2D)           (None, 16, 16, 32)        50208     
_________________________________________________________________
batch_normalization_7 (Batch (None, 16, 16, 32)        128       
_________________________________________________________________
activation_7 (Activation)    (None, 16, 16, 32)        0         
_________________________________________________________________
max_pooling2d_6 (MaxPooling2 (None, 8, 8, 32)          0         
_________________________________________________________________
conv2d_15 (Conv2D)           (None, 8, 8, 64)          18496     
_________________________________________________________________
conv2d_16 (Conv2D)           (None, 8, 8, 64)          36928     
_________________________________________________________________
batch_normalization_8 (Batch (None, 8, 8, 64)          256       
_________________________________________________________________
activation_8 (Activation)    (None, 8, 8, 64)          0         
_________________________________________________________________
average_pooling2d_2 (Average (None, 4, 4, 64)          0         
_________________________________________________________________
flatten_2 (Flatten)          (None, 1024)              0         
_________________________________________________________________
dense_3 (Dense)              (None, 1024)              1049600   
_________________________________________________________________
dropout_2 (Dropout)          (None, 1024)              0         
_________________________________________________________________
dense_4 (Dense)              (None, 2)                 2050      
=================================================================
Total params: 1,290,922
Trainable params: 1,290,682
Non-trainable params: 240

epoch 0
Epoch 1/1
18/18 [==============================] - 158s 9s/step - loss: 0.8175 - categorical_accuracy: 0.5300
200/200 [==============================] - 6s 30ms/step
categorical_accuracy: 61.50%
epoch 1
Epoch 1/1
18/18 [==============================] - 156s 9s/step - loss: 0.7062 - categorical_accuracy: 0.5667
200/200 [==============================] - 6s 29ms/step
categorical_accuracy: 61.50%
epoch 2
Epoch 1/1
18/18 [==============================] - 156s 9s/step - loss: 0.4627 - categorical_accuracy: 0.7811
200/200 [==============================] - 6s 29ms/step
categorical_accuracy: 86.50%
epoch 3
Epoch 1/1
18/18 [==============================] - 156s 9s/step - loss: 0.1314 - categorical_accuracy: 0.9611
200/200 [==============================] - 6s 29ms/step
categorical_accuracy: 85.50%
epoch 4
Epoch 1/1
18/18 [==============================] - 156s 9s/step - loss: 0.0670 - categorical_accuracy: 0.9817
200/200 [==============================] - 6s 29ms/step
categorical_accuracy: 39.00%
epoch 5
Epoch 1/1
18/18 [==============================] - 156s 9s/step - loss: 0.0177 - categorical_accuracy: 0.9994
200/200 [==============================] - 6s 29ms/step
categorical_accuracy: 39.00%

[geoffwoollard]

Note that model.evaluate can be misleading and one should use model.predict_proba with an applied cutoff instead (write custom accuracy function). I'm not sure why, but suspect that it learns a non-general cutoff between [0,1] from fitting and then applies that to the validation data. The roc auc are good on the validation data, but the categorical_accuracy is poor.

from sklearn.metrics import roc_curve, auc
from keras.models import model_from_yaml
from keras.optimizers import SGD

tprs={}
fprs={}
thresholds={}
roc_aucs={}
for epoch,fname in enumerate(glob.glob('models/model-sc-nt0.4-J159-J160-0.77Mparams-ep-*yaml')):
  print(fname)
  # load YAML and create model
  with open(fname, 'r') as yaml_file:
    loaded_model_yaml = yaml_file.read()
  #yaml_file.close()
  loaded_model = model_from_yaml(loaded_model_yaml)
  # load weights into new model
  loaded_model.load_weights(fname.replace('.yaml',".h5"))
  loaded_model.compile(optimizer = SGD() , loss='categorical_crossentropy', metrics=['categorical_accuracy'])
  y_score = loaded_model.predict_proba(X_val)[:,0]
  fprs[epoch], tprs[epoch], thresholds[epoch] = roc_curve(y_true = Y_val[:,0],y_score=y_score)
  roc_aucs[epoch] = auc(fprs[epoch], tprs[epoch])
  print('auc=%.2f'%roc_aucs[epoch])
  scores = loaded_model.evaluate(X_val, Y_val)
  print("%s: %.2f%%" % (loaded_model.metrics_names[1], scores[1] * 100))

models/model-sc-nt0.4-J159-J160-0.77Mparams-ep-0-20190329-2226.yaml
auc=0.59
200/200 [==============================] - 6s 31ms/step
categorical_accuracy: 59.50%
models/model-sc-nt0.4-J159-J160-0.77Mparams-ep-1-20190329-2229.yaml
auc=0.65
200/200 [==============================] - 6s 31ms/step
categorical_accuracy: 57.00%
models/model-sc-nt0.4-J159-J160-0.77Mparams-ep-2-20190329-2231.yaml
auc=0.73
200/200 [==============================] - 6s 31ms/step
categorical_accuracy: 57.00%
models/model-sc-nt0.4-J159-J160-0.77Mparams-ep-3-20190329-2234.yaml
auc=0.91
200/200 [==============================] - 6s 31ms/step
categorical_accuracy: 57.00%
models/model-sc-nt0.4-J159-J160-0.77Mparams-ep-4-20190329-2237.yaml
auc=0.97
200/200 [==============================] - 6s 31ms/step
categorical_accuracy: 57.00%
models/model-sc-nt0.4-J159-J160-0.77Mparams-ep-5-20190329-2240.yaml
auc=0.97
200/200 [==============================] - 6s 31ms/step
categorical_accuracy: 57.00%
models/model-sc-nt0.4-J159-J160-0.77Mparams-ep-6-20190329-2242.yaml
auc=0.97
200/200 [==============================] - 6s 31ms/step
categorical_accuracy: 57.00%
models/model-sc-nt0.4-J159-J160-0.77Mparams-ep-7-20190329-2245.yaml
auc=0.95
200/200 [==============================] - 6s 31ms/step
categorical_accuracy: 57.00%
models/model-sc-nt0.4-J159-J160-0.77Mparams-ep-8-20190329-2248.yaml
auc=0.95
200/200 [==============================] - 6s 32ms/step
categorical_accuracy: 57.00%
models/model-sc-nt0.4-J159-J160-0.77Mparams-ep-9-20190329-2250.yaml
auc=0.90
200/200 [==============================] - 6s 32ms/step
categorical_accuracy: 57.00%

[geoffwoollard]

The model is unable to generalize at a SNR=0.1

model = deepconsensus_layers_wrapper(
    input_shape=X_val.shape[1::],
    num_hidden_layers=4,

    conv2d1_k=(30,15,7,3),
    conv2d2_k=(30,15,7,3),
    conv2d1_n=(8,8,32,64),
    conv2d2_n=(8,16,32,64),
    mp_k=(3,3,3,4),
    mp_strides=(4,4,2,2),
  pooling_type=('max','max','max','av'),
  dense13_n=512,
    dropout13_rate=0.25
)

It can train perfectly, but gets up to 0.7 auc on the validation data

Now the architecture can be changed, with the goal of improving the generalizability.

for epoch in range(10):
    print('epoch %i' % epoch)
    model.fit_generator(fit_generator_helper.image_loader(df_train,batch_size=batch_size,nx=nx,ny=ny,crop_n=256),
                    steps_per_epoch=steps_per_epoch, # steps_per_epoch is number of batches per epoch
                    epochs=1,
                   )
    scores = model.evaluate(X_val, Y_val)
    print("%s: %.2f%%" % (model.metrics_names[1], scores[1] * 100))

    roc_aucs[epoch],fprs[epoch], tprs[epoch], thresholds[epoch] = roc_auc(X_val,Y_val,model)
    print('auc=%.2f'%roc_aucs[epoch])

    title='models/model-sc-nt0.1-J159-J160-0.77Mparams-ep-%s-' % epoch
    model_yaml = model.to_yaml()
    with open(title+timestr+'.yaml', "w") as yaml_file:
        yaml_file.write(model_yaml)
    model.save_weights(title+timestr+".h5")

poch 0
Epoch 1/1
18/18 [==============================] - 207s 11s/step - loss: 0.7699 - categorical_accuracy: 0.5172
200/200 [==============================] - 6s 29ms/step
categorical_accuracy: 46.50%
auc=0.49
epoch 1
Epoch 1/1
18/18 [==============================] - 150s 8s/step - loss: 0.5590 - categorical_accuracy: 0.7039
200/200 [==============================] - 6s 28ms/step
categorical_accuracy: 46.50%
auc=0.50
epoch 2
Epoch 1/1
18/18 [==============================] - 150s 8s/step - loss: 0.4494 - categorical_accuracy: 0.7989
200/200 [==============================] - 6s 28ms/step
categorical_accuracy: 46.50%
auc=0.50
epoch 3
Epoch 1/1
18/18 [==============================] - 150s 8s/step - loss: 0.2728 - categorical_accuracy: 0.8933
200/200 [==============================] - 6s 28ms/step
categorical_accuracy: 53.50%
auc=0.69
epoch 4
Epoch 1/1
18/18 [==============================] - 150s 8s/step - loss: 0.1961 - categorical_accuracy: 0.9344
200/200 [==============================] - 6s 28ms/step
categorical_accuracy: 53.50%
auc=0.53
epoch 5
Epoch 1/1
18/18 [==============================] - 150s 8s/step - loss: 0.1105 - categorical_accuracy: 0.9622
200/200 [==============================] - 6s 28ms/step
categorical_accuracy: 53.50%
auc=0.65
epoch 6
Epoch 1/1
18/18 [==============================] - 150s 8s/step - loss: 0.0141 - categorical_accuracy: 0.9994
200/200 [==============================] - 6s 28ms/step
categorical_accuracy: 53.50%
auc=0.67
epoch 7
Epoch 1/1
18/18 [==============================] - 150s 8s/step - loss: 0.0067 - categorical_accuracy: 1.0000
200/200 [==============================] - 6s 28ms/step
categorical_accuracy: 53.50%
auc=0.70
epoch 8
Epoch 1/1
18/18 [==============================] - 150s 8s/step - loss: 0.0049 - categorical_accuracy: 1.0000
200/200 [==============================] - 6s 28ms/step
categorical_accuracy: 53.50%
auc=0.71
epoch 9
Epoch 1/1
18/18 [==============================] - 150s 8s/step - loss: 0.0039 - categorical_accuracy: 1.0000
200/200 [==============================] - 6s 28ms/step
categorical_accuracy: 53.50%
auc=0.69

[geoffwoollard]

We can also change the kernel size to be 15x15 initially, and the pooling kernels to be smaller. We still get perfect performance eventually

model = deepconsensus_layers_wrapper(
    input_shape=X_val.shape[1::],
    num_hidden_layers=4,

    conv2d1_k=(15,15,7,3),
    conv2d2_k=(15,15,7,3),
    conv2d1_n=(8,8,32,64),
    conv2d2_n=(8,16,32,64),
    mp_k=(3,3,3,4),
    mp_strides=(4,4,2,2),
  pooling_type=('max','max','max','av'),
  dense13_n=512,
    dropout13_rate=0.5
)
model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_81 (Conv2D)           (None, 256, 256, 8)       1808      
_________________________________________________________________
conv2d_82 (Conv2D)           (None, 256, 256, 8)       14408     
_________________________________________________________________
batch_normalization_41 (Batc (None, 256, 256, 8)       32        
_________________________________________________________________
activation_41 (Activation)   (None, 256, 256, 8)       0         
_________________________________________________________________
max_pooling2d_31 (MaxPooling (None, 64, 64, 8)         0         
_________________________________________________________________
conv2d_83 (Conv2D)           (None, 64, 64, 8)         14408     
_________________________________________________________________
conv2d_84 (Conv2D)           (None, 64, 64, 16)        28816     
_________________________________________________________________
batch_normalization_42 (Batc (None, 64, 64, 16)        64        
_________________________________________________________________
activation_42 (Activation)   (None, 64, 64, 16)        0         
_________________________________________________________________
max_pooling2d_32 (MaxPooling (None, 16, 16, 16)        0         
_________________________________________________________________
conv2d_85 (Conv2D)           (None, 16, 16, 32)        25120     
_________________________________________________________________
conv2d_86 (Conv2D)           (None, 16, 16, 32)        50208     
_________________________________________________________________
batch_normalization_43 (Batc (None, 16, 16, 32)        128       
_________________________________________________________________
activation_43 (Activation)   (None, 16, 16, 32)        0         
_________________________________________________________________
max_pooling2d_33 (MaxPooling (None, 8, 8, 32)          0         
_________________________________________________________________
conv2d_87 (Conv2D)           (None, 8, 8, 64)          18496     
_________________________________________________________________
conv2d_88 (Conv2D)           (None, 8, 8, 64)          36928     
_________________________________________________________________
batch_normalization_44 (Batc (None, 8, 8, 64)          256       
_________________________________________________________________
activation_44 (Activation)   (None, 8, 8, 64)          0         
_________________________________________________________________
average_pooling2d_11 (Averag (None, 4, 4, 64)          0         
_________________________________________________________________
flatten_11 (Flatten)         (None, 1024)              0         
_________________________________________________________________
dense_21 (Dense)             (None, 512)               524800    
_________________________________________________________________
dropout_11 (Dropout)         (None, 512)               0         
_________________________________________________________________
dense_22 (Dense)             (None, 2)                 1026      
=================================================================
Total params: 716,498
Trainable params: 716,258
Non-trainable params: 240

epoch 0
Epoch 1/1
18/18 [==============================] - 31s 2s/step - loss: 0.8407 - categorical_accuracy: 0.4872
200/200 [==============================] - 2s 12ms/step
categorical_accuracy: 46.50%
auc=0.67
epoch 1
Epoch 1/1
18/18 [==============================] - 22s 1s/step - loss: 0.6893 - categorical_accuracy: 0.5794
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 46.50%
auc=0.82
epoch 2
Epoch 1/1
18/18 [==============================] - 22s 1s/step - loss: 0.4071 - categorical_accuracy: 0.8150
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 46.50%
auc=0.50
epoch 3
Epoch 1/1
18/18 [==============================] - 22s 1s/step - loss: 0.2496 - categorical_accuracy: 0.9022
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 46.50%
auc=0.50
epoch 4
Epoch 1/1
18/18 [==============================] - 22s 1s/step - loss: 0.1718 - categorical_accuracy: 0.9383
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 46.50%
auc=0.50
epoch 5
Epoch 1/1
18/18 [==============================] - 22s 1s/step - loss: 0.0949 - categorical_accuracy: 0.9772
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 82.50%
auc=0.96
epoch 6
Epoch 1/1
18/18 [==============================] - 22s 1s/step - loss: 0.0786 - categorical_accuracy: 0.9772
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 83.50%
auc=0.96
epoch 7
Epoch 1/1
18/18 [==============================] - 22s 1s/step - loss: 0.0166 - categorical_accuracy: 1.0000
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 53.50%
auc=0.96
epoch 8
Epoch 1/1
18/18 [==============================] - 22s 1s/step - loss: 0.0099 - categorical_accuracy: 1.0000
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 53.50%
auc=0.95
epoch 9
Epoch 1/1
18/18 [==============================] - 22s 1s/step - loss: 0.0073 - categorical_accuracy: 1.0000
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 53.50%
auc=0.94

[geoffwoollard]

Even at snr=0.1 and a deeper network with an extra block, we can achieve performance, but get overfitting. Eventually the roc is inverted, even for the training data. This is confusing since the loss is decreasing and the categorical_accuracy is ideal.

model = deepconsensus_layers_wrapper(
    input_shape=X_val.shape[1::],
    num_hidden_layers=5,

    conv2d1_k=(15,15,9,7,3),
    conv2d2_k=(15,15,9,7,3),
    conv2d1_n=(8,8,8,32,64),
    conv2d2_n=(8,16,16,32,64),
    mp_k=(3,3,3,3,4),
    mp_strides=(2,2,2,2,2),
  pooling_type=('max','max','max','max','av'),
  dense13_n=512,
    dropout13_rate=0.5
)
model.summary()

Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 256, 256, 8)       1808      
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 256, 256, 8)       14408     
_________________________________________________________________
batch_normalization_1 (Batch (None, 256, 256, 8)       32        
_________________________________________________________________
activation_1 (Activation)    (None, 256, 256, 8)       0         
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 128, 128, 8)       0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 128, 128, 8)       14408     
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 128, 128, 16)      28816     
_________________________________________________________________
batch_normalization_2 (Batch (None, 128, 128, 16)      64        
_________________________________________________________________
activation_2 (Activation)    (None, 128, 128, 16)      0         
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 64, 64, 16)        0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 64, 64, 8)         10376     
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 64, 64, 16)        10384     
_________________________________________________________________
batch_normalization_3 (Batch (None, 64, 64, 16)        64        
_________________________________________________________________
activation_3 (Activation)    (None, 64, 64, 16)        0         
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 32, 32, 16)        0         
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 32, 32, 32)        25120     
_________________________________________________________________
conv2d_8 (Conv2D)            (None, 32, 32, 32)        50208     
_________________________________________________________________
batch_normalization_4 (Batch (None, 32, 32, 32)        128       
_________________________________________________________________
activation_4 (Activation)    (None, 32, 32, 32)        0         
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 16, 16, 32)        0         
_________________________________________________________________
conv2d_9 (Conv2D)            (None, 16, 16, 64)        18496     
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 16, 16, 64)        36928     
_________________________________________________________________
batch_normalization_5 (Batch (None, 16, 16, 64)        256       
_________________________________________________________________
activation_5 (Activation)    (None, 16, 16, 64)        0         
_________________________________________________________________
average_pooling2d_1 (Average (None, 8, 8, 64)          0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 4096)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 512)               2097664   
_________________________________________________________________
dropout_1 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 2)                 1026      
=================================================================
Total params: 2,310,186
Trainable params: 2,309,914
Non-trainable params: 272

title_stem = 'models/model-sc-nt0.1-J159-J160-256pix-5hl-2.3Mparams-ep-%s-'
for epoch in range(0,2):
    print('epoch %i' % epoch)
    model.fit_generator(fit_generator_helper.image_loader(df_train,batch_size=batch_size,nx=nx,ny=ny,crop_n=crop_n),
                    steps_per_epoch=steps_per_epoch, # steps_per_epoch is number of batches per epoch
                    epochs=1,
                   )
    roc_auc_train,_,_,_ = roc_auc(X_train,Y_train,model)
    print('train auc=%.2f'%roc_auc_train)

    scores = model.evaluate(X_val, Y_val)
    print("%s: %.2f%%" % (model.metrics_names[1], scores[1] * 100))

    roc_aucs[epoch],fprs[epoch], tprs[epoch], thresholds[epoch] = roc_auc(X_val,Y_val,model)
    print('val auc=%.2f'%roc_aucs[epoch])

    title=title_stem % epoch
    model_yaml = model.to_yaml()
    with open(title+timestr+'.yaml', "w") as yaml_file:
        yaml_file.write(model_yaml)
    model.save_weights(title+timestr+".h5")

epoch 2
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.5139 - categorical_accuracy: 0.7589
train auc=0.50
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 51.50%
val auc=0.50
epoch 3
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.3261 - categorical_accuracy: 0.8567
train auc=0.50
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 51.50%
val auc=0.50
epoch 4
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.2569 - categorical_accuracy: 0.8889
train auc=0.50
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 51.50%
val auc=0.50
epoch 5
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.1901 - categorical_accuracy: 0.9283
train auc=0.99
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 51.50%
val auc=0.96
epoch 6
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.1294 - categorical_accuracy: 0.9567
train auc=0.72
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.77
epoch 7
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0714 - categorical_accuracy: 0.9839
train auc=0.64
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.66
epoch 8
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0404 - categorical_accuracy: 0.9933
train auc=0.10
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.18
epoch 9
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0144 - categorical_accuracy: 0.9994
train auc=0.07
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.15
epoch 10
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0082 - categorical_accuracy: 1.0000
train auc=0.05
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.12
epoch 11
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0056 - categorical_accuracy: 1.0000
train auc=0.03
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.10
epoch 12
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0042 - categorical_accuracy: 1.0000
train auc=0.02
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.10
epoch 13
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0033 - categorical_accuracy: 1.0000
train auc=0.02
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.09
epoch 14
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0029 - categorical_accuracy: 1.0000
train auc=0.03
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.09
epoch 15
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0024 - categorical_accuracy: 1.0000
train auc=0.04
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.11
epoch 16
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0023 - categorical_accuracy: 1.0000
train auc=0.06
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.13
epoch 17
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0018 - categorical_accuracy: 1.0000
train auc=0.12
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.20
epoch 18
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0017 - categorical_accuracy: 1.0000
train auc=0.36
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.44
epoch 19
Epoch 1/1
18/18 [==============================] - 24s 1s/step - loss: 0.0015 - categorical_accuracy: 1.0000
train auc=0.77
200/200 [==============================] - 1s 4ms/step
categorical_accuracy: 48.50%
val auc=0.78

[geoffwoollard]

It we just use the default of 4 hidden layers we can get near perfect performance (0.95 auc validation). We also don't end up overfitting.

model = deepconsensus_layers_wrapper(
    input_shape=X_val.shape[1::],
    num_hidden_layers=4,

    conv2d1_k=(15,9,7,3),
    conv2d2_k=(15,9,7,3),
    conv2d1_n=(8,8,32,64),
    conv2d2_n=(8,16,32,64),
    mp_k=(3,3,3,4),
    mp_strides=(4,2,2,2),
  pooling_type=('max','max','max','av'),
  dense13_n=512,
    dropout13_rate=0.5
)
model.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_55 (Conv2D)           (None, 256, 256, 8)       1808      
_________________________________________________________________
conv2d_56 (Conv2D)           (None, 256, 256, 8)       14408     
_________________________________________________________________
batch_normalization_28 (Batc (None, 256, 256, 8)       32        
_________________________________________________________________
activation_28 (Activation)   (None, 256, 256, 8)       0         
_________________________________________________________________
max_pooling2d_22 (MaxPooling (None, 64, 64, 8)         0         
_________________________________________________________________
conv2d_57 (Conv2D)           (None, 64, 64, 8)         5192      
_________________________________________________________________
conv2d_58 (Conv2D)           (None, 64, 64, 16)        10384     
_________________________________________________________________
batch_normalization_29 (Batc (None, 64, 64, 16)        64        
_________________________________________________________________
activation_29 (Activation)   (None, 64, 64, 16)        0         
_________________________________________________________________
max_pooling2d_23 (MaxPooling (None, 32, 32, 16)        0         
_________________________________________________________________
conv2d_59 (Conv2D)           (None, 32, 32, 32)        25120     
_________________________________________________________________
conv2d_60 (Conv2D)           (None, 32, 32, 32)        50208     
_________________________________________________________________
batch_normalization_30 (Batc (None, 32, 32, 32)        128       
_________________________________________________________________
activation_30 (Activation)   (None, 32, 32, 32)        0         
_________________________________________________________________
max_pooling2d_24 (MaxPooling (None, 16, 16, 32)        0         
_________________________________________________________________
conv2d_61 (Conv2D)           (None, 16, 16, 64)        18496     
_________________________________________________________________
conv2d_62 (Conv2D)           (None, 16, 16, 64)        36928     
_________________________________________________________________
batch_normalization_31 (Batc (None, 16, 16, 64)        256       
_________________________________________________________________
activation_31 (Activation)   (None, 16, 16, 64)        0         
_________________________________________________________________
average_pooling2d_7 (Average (None, 8, 8, 64)          0         
_________________________________________________________________
flatten_7 (Flatten)          (None, 4096)              0         
_________________________________________________________________
dense_13 (Dense)             (None, 512)               2097664   
_________________________________________________________________
dropout_7 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_14 (Dense)             (None, 2)                 1026      
=================================================================
Total params: 2,261,714
Trainable params: 2,261,474
Non-trainable params: 240
_________________________________________________________________

timestr = time.strftime('%Y%m%d-%H%M')
title_stem = 'models/model-sc-nt0.1-J159-J160-256pix-4hl-2.3Mparams-ep-%s-'
for epoch in range(0,2):
    print('epoch %i' % epoch)
    model.fit_generator(fit_generator_helper.image_loader(df_train,batch_size=batch_size,nx=nx,ny=ny,crop_n=crop_n),
                    steps_per_epoch=steps_per_epoch, # steps_per_epoch is number of batches per epoch
                    epochs=1,
                   )
    roc_auc_train,_,_,_ = roc_auc(X_train,Y_train,model)
    print('train auc=%.2f'%roc_auc_train)

    scores = model.evaluate(X_val, Y_val)
    print("%s: %.2f%%" % (model.metrics_names[1], scores[1] * 100))

    roc_aucs[epoch],fprs[epoch], tprs[epoch], thresholds[epoch] = roc_auc(X_val,Y_val,model)
    print('val auc=%.2f'%roc_aucs[epoch])

    title=title_stem % epoch
    model_yaml = model.to_yaml()
    with open(title+timestr+'.yaml', "w") as yaml_file:
        yaml_file.write(model_yaml)
    model.save_weights(title+timestr+".h5")

epoch 0
Epoch 1/1
18/18 [==============================] - 23s 1s/step - loss: 1.0355 - categorical_accuracy: 0.5161
train auc=0.68
200/200 [==============================] - 1s 6ms/step
categorical_accuracy: 51.50%
val auc=0.61
epoch 1
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.6960 - categorical_accuracy: 0.5806
train auc=0.50
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 51.50%
val auc=0.50
epoch 2
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.4369 - categorical_accuracy: 0.8011
train auc=0.50
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 51.50%
val auc=0.50
epoch 3
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.2918 - categorical_accuracy: 0.8733
train auc=0.50
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 51.50%
val auc=0.50
epoch 4
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.1821 - categorical_accuracy: 0.9389
train auc=0.50
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 51.50%
val auc=0.50
epoch 5
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.1322 - categorical_accuracy: 0.9594
train auc=0.91
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 51.50%
val auc=0.80
epoch 6
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0467 - categorical_accuracy: 0.9944
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 76.00%
val auc=0.95
epoch 7
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0193 - categorical_accuracy: 0.9989
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 70.50%
val auc=0.95
epoch 8
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0100 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 78.50%
val auc=0.94
epoch 9
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0063 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 50.00%
val auc=0.95
epoch 10
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0052 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 48.50%
val auc=0.95
epoch 11
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0038 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 48.50%
val auc=0.95
epoch 12
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0030 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 48.50%
val auc=0.95
epoch 13
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0029 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 48.50%
val auc=0.95
epoch 14
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0027 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 48.50%
val auc=0.95
epoch 15
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0022 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 48.50%
val auc=0.95
epoch 16
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0018 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 48.50%
val auc=0.95
epoch 17
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0018 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 48.50%
val auc=0.95
epoch 18
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0016 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 49.00%
val auc=0.95
epoch 19
Epoch 1/1
18/18 [==============================] - 21s 1s/step - loss: 0.0015 - categorical_accuracy: 1.0000
train auc=1.00
200/200 [==============================] - 1s 3ms/step
categorical_accuracy: 49.00%
val auc=0.95

[geoffwoollard]

We can make the model even simpler for snr=0.1 and get ideal performance (dense layer 256). It comes in on epoch 6, same as with dense layer 512.

model = deepconsensus_layers_wrapper(
    input_shape=X_val.shape[1::],
    num_hidden_layers=4,

    conv2d1_k=(15,9,7,3),
    conv2d2_k=(15,9,7,3),
    conv2d1_n=(8,8,32,64),
    conv2d2_n=(8,16,32,64),
    mp_k=(3,3,3,4),
    mp_strides=(4,2,2,2),
  pooling_type=('max','max','max','av'),
  dense13_n=256,
    dropout13_rate=0.5
)
model.summary()

Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_63 (Conv2D)           (None, 256, 256, 8)       1808      
_________________________________________________________________
conv2d_64 (Conv2D)           (None, 256, 256, 8)       14408     
_________________________________________________________________
batch_normalization_32 (Batc (None, 256, 256, 8)       32        
_________________________________________________________________
activation_32 (Activation)   (None, 256, 256, 8)       0         
_________________________________________________________________
max_pooling2d_25 (MaxPooling (None, 64, 64, 8)         0         
_________________________________________________________________
conv2d_65 (Conv2D)           (None, 64, 64, 8)         5192      
_________________________________________________________________
conv2d_66 (Conv2D)           (None, 64, 64, 16)        10384     
_________________________________________________________________
batch_normalization_33 (Batc (None, 64, 64, 16)        64        
_________________________________________________________________
activation_33 (Activation)   (None, 64, 64, 16)        0         
_________________________________________________________________
max_pooling2d_26 (MaxPooling (None, 32, 32, 16)        0         
_________________________________________________________________
conv2d_67 (Conv2D)           (None, 32, 32, 32)        25120     
_________________________________________________________________
conv2d_68 (Conv2D)           (None, 32, 32, 32)        50208     
_________________________________________________________________
batch_normalization_34 (Batc (None, 32, 32, 32)        128       
_________________________________________________________________
activation_34 (Activation)   (None, 32, 32, 32)        0         
_________________________________________________________________
max_pooling2d_27 (MaxPooling (None, 16, 16, 32)        0         
_________________________________________________________________
conv2d_69 (Conv2D)           (None, 16, 16, 64)        18496     
_________________________________________________________________
conv2d_70 (Conv2D)           (None, 16, 16, 64)        36928     
_________________________________________________________________
batch_normalization_35 (Batc (None, 16, 16, 64)        256       
_________________________________________________________________
activation_35 (Activation)   (None, 16, 16, 64)        0         
_________________________________________________________________
average_pooling2d_8 (Average (None, 8, 8, 64)          0         
_________________________________________________________________
flatten_8 (Flatten)          (None, 4096)              0         
_________________________________________________________________
dense_15 (Dense)             (None, 256)               1048832   
_________________________________________________________________
dropout_8 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_16 (Dense)             (None, 2)                 514       
=================================================================
Total params: 1,212,370
Trainable params: 1,212,130
Non-trainable params: 240
_________________________________________________________________
title_stem = 'models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-%s-'
for epoch in range(0,20):
    #10 print('epoch %i' % epoch)
    model.fit_generator(fit_generator_helper.image_loader(df_train,batch_size=batch_size,nx=nx,ny=ny,crop_n=crop_n),
                    steps_per_epoch=steps_per_epoch, # steps_per_epoch is number of batches per epoch
                    epochs=1,
                   )
    roc_auc_train,_,_,_ = roc_auc(X_train,Y_train,model)
    #print('train auc=%.2f'%roc_auc_train)

    scores = model.evaluate(X_val, Y_val)
    #print("%s: %.2f%%" % (model.metrics_names[1], scores[1] * 100))

    roc_aucs[epoch],fprs[epoch], tprs[epoch], thresholds[epoch] = roc_auc(X_val,Y_val,model)
    #print('val auc=%.2f'%roc_aucs[epoch])

    title=title_stem % epoch
    model_yaml = model.to_yaml()
    with open(title+timestr+'.yaml', "w") as yaml_file:
        yaml_file.write(model_yaml)
    model.save_weights(title+timestr+".h5")

tprs, fprs, thresholds, roc_aucs={}, {}, {}, {}

for epoch,fname in enumerate(glob.glob('models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-*yaml')):
  print(fname)
  # load YAML and create model
  with open(fname, 'r') as yaml_file:
    loaded_model_yaml = yaml_file.read()
  #yaml_file.close()
  loaded_model = model_from_yaml(loaded_model_yaml)
  # load weights into new model
  loaded_model.load_weights(fname.replace('.yaml',".h5"))
  loaded_model.compile(optimizer = SGD() , loss='categorical_crossentropy', metrics=['categorical_accuracy'])
  roc_auc_train,_,_,_ = roc_auc(X_train,Y_train,loaded_model)
  print('tain auc=%.2f'%roc_auc_train)

  roc_aucs[epoch],fprs[epoch], tprs[epoch], thresholds[epoch] = roc_auc(X_val,Y_val,loaded_model)
  print('val auc=%.2f'%roc_aucs[epoch])
  scores = loaded_model.evaluate(X_val, Y_val)
  print("%s: %.2f%%" % (loaded_model.metrics_names[1], scores[1] * 100))

models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-0-20190402-1535.yaml
tain auc=0.57
val auc=0.64
200/200 [==============================] - 2s 9ms/step
categorical_accuracy: 47.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-1-20190402-1535.yaml
tain auc=0.50
val auc=0.50
200/200 [==============================] - 2s 9ms/step
categorical_accuracy: 47.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-2-20190402-1535.yaml
tain auc=0.50
val auc=0.50
200/200 [==============================] - 2s 10ms/step
categorical_accuracy: 47.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-3-20190402-1535.yaml
tain auc=0.50
val auc=0.50
200/200 [==============================] - 2s 10ms/step
categorical_accuracy: 47.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-4-20190402-1535.yaml
tain auc=0.50
val auc=0.50
200/200 [==============================] - 2s 10ms/step
categorical_accuracy: 47.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-5-20190402-1535.yaml
tain auc=0.50
val auc=0.50
200/200 [==============================] - 2s 11ms/step
categorical_accuracy: 47.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-6-20190402-1535.yaml
tain auc=0.99
val auc=1.00
200/200 [==============================] - 2s 11ms/step
categorical_accuracy: 47.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-7-20190402-1535.yaml
tain auc=0.99
val auc=1.00
200/200 [==============================] - 2s 11ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-8-20190402-1535.yaml
tain auc=0.99
val auc=1.00
200/200 [==============================] - 2s 12ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-9-20190402-1535.yaml
tain auc=1.00
val auc=1.00
200/200 [==============================] - 2s 12ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-10-20190402-1535.yaml
tain auc=1.00
val auc=1.00
200/200 [==============================] - 3s 13ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-11-20190402-1535.yaml
tain auc=1.00
val auc=1.00
200/200 [==============================] - 3s 13ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-12-20190402-1535.yaml
tain auc=1.00
val auc=1.00
200/200 [==============================] - 3s 13ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-13-20190402-1535.yaml
tain auc=1.00
val auc=1.00
200/200 [==============================] - 3s 14ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-14-20190402-1535.yaml
tain auc=1.00
val auc=1.00
200/200 [==============================] - 3s 14ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-15-20190402-1535.yaml
tain auc=1.00
val auc=1.00
200/200 [==============================] - 3s 15ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-16-20190402-1535.yaml
tain auc=1.00
val auc=1.00
200/200 [==============================] - 3s 15ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-17-20190402-1535.yaml
tain auc=1.00
val auc=1.00
200/200 [==============================] - 3s 15ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-18-20190402-1535.yaml
tain auc=1.00
val auc=1.00
200/200 [==============================] - 3s 16ms/step
categorical_accuracy: 52.50%
models/model-sc-nt0.1-J159-J160-256pix-256dense-1.2Mparams-ep-19-20190402-1535.yaml
tain auc=1.00
val auc=1.00
200/200 [==============================] - 3s 16ms/step
categorical_accuracy: 52.50%

[Davjes15]

@geoffwoollard could you please share J159 and J160 I would like to run the model on those images but I just realize that I dont have those images.

[Davjes15]

@geoffwoollard Ok I will use 153/154 Thank you