Open sdtaylor opened 3 years ago
same as above but going another 20 epochs w/ lr at 0.001
precision recall f1-score support
unknown 0.67 0.71 0.69 14
vegetation 0.89 0.95 0.92 793
residue 0.77 0.65 0.70 293
soil 0.67 0.64 0.65 144
snow 0.94 0.73 0.82 82
water 0.38 0.88 0.53 16
accuracy 0.83 1342
macro avg 0.72 0.76 0.72 1342
weighted avg 0.84 0.83 0.83 1342
class_description unknown vegetation residue soil snow water
class_description
unknown 10 2 1 0 0 1
vegetation 0 752 23 13 2 3
residue 1 62 190 32 2 6
soil 0 24 25 92 0 3
snow 4 1 6 1 60 10
water 0 0 2 0 0 14
precision recall f1-score support
uknown 0.67 0.71 0.69 14
unknown_plant 0.80 0.61 0.70 327
large_grass 0.72 0.93 0.81 238
small_grass 0.83 0.67 0.74 258
other 0.53 0.72 0.61 188
fallow 0.68 0.87 0.76 15
none 0.86 0.81 0.83 302
accuracy 0.74 1342
macro avg 0.73 0.76 0.74 1342
weighted avg 0.77 0.74 0.75 1342
class_description uknown unknown_plant large_grass small_grass other fallow none
class_description
uknown 10 1 0 0 0 0 3
unknown_plant 0 201 36 10 47 4 29
large_grass 0 7 222 3 6 0 0
small_grass 0 8 20 173 54 0 3
other 0 8 28 12 136 0 4
fallow 0 0 0 0 1 13 1
none 5 25 4 10 12 2 244
precision recall f1-score support
unknown 0.67 0.71 0.69 14
emergence 0.71 0.54 0.61 190
growth 0.80 0.89 0.84 438
flowers 0.74 0.86 0.80 160
senescing 0.61 0.62 0.62 138
senesced 0.66 0.63 0.65 100
no_crop 0.89 0.81 0.85 302
accuracy 0.77 1342
macro avg 0.73 0.72 0.72 1342
weighted avg 0.77 0.77 0.77 1342
class_description unknown emergence growth flowers senescing senesced no_crop
class_description
unknown 10 0 1 0 0 0 3
emergence 0 102 62 1 3 3 19
growth 0 20 391 19 7 0 1
flowers 0 0 12 137 11 0 0
senescing 0 0 13 26 86 12 1
senesced 0 2 2 1 25 63 7
no_crop 5 20 8 0 8 17 244
starting again from 55 epochs and only going 5 more with lr 0.001
this version: https://github.com/sdtaylor/PhenocamCNN2/blob/15594755e01b60ffb2b13521865a878c7fbb51f1/test_cnn_basic.py
this model saved in vgg16_v1_70epochs.h5
precision recall f1-score support
unknown 0.40 0.44 0.42 9
vegetation 0.90 0.94 0.92 776
residue 0.75 0.72 0.73 296
soil 0.72 0.62 0.66 166
snow 0.95 0.78 0.85 72
water 0.53 0.78 0.63 23
accuracy 0.84 1342
macro avg 0.71 0.71 0.70 1342
weighted avg 0.84 0.84 0.84 1342
class_description unknown vegetation residue soil snow water
class_description
unknown 4 3 0 0 2 0
vegetation 1 732 29 12 0 2
residue 0 51 212 29 1 3
soil 0 23 36 103 0 4
snow 4 1 4 0 56 7
water 1 2 2 0 0 18
precision recall f1-score support
uknown 0.44 0.44 0.44 9
unknown_plant 0.85 0.76 0.80 331
large_grass 0.80 0.94 0.86 259
small_grass 0.78 0.66 0.71 235
other 0.61 0.73 0.66 197
fallow 0.52 0.92 0.67 12
none 0.91 0.84 0.88 299
accuracy 0.79 1342
macro avg 0.70 0.76 0.72 1342
weighted avg 0.80 0.79 0.79 1342
class_description uknown unknown_plant large_grass small_grass other fallow none
class_description
uknown 4 1 0 2 0 0 2
unknown_plant 0 251 25 13 24 4 14
large_grass 0 9 244 3 3 0 0
small_grass 1 10 11 154 54 0 5
other 0 9 20 22 143 0 3
fallow 0 0 0 0 0 11 1
none 4 17 6 4 10 6 252
precision recall f1-score support
unknown 0.36 0.44 0.40 9
emergence 0.81 0.63 0.71 199
growth 0.79 0.92 0.85 414
flowers 0.70 0.80 0.75 150
senescing 0.69 0.69 0.69 168
senesced 0.69 0.61 0.65 103
no_crop 0.92 0.80 0.86 299
accuracy 0.78 1342
macro avg 0.71 0.70 0.70 1342
weighted avg 0.79 0.78 0.78 1342
class_description unknown emergence growth flowers senescing senesced no_crop
class_description
unknown 4 0 1 1 1 0 2
emergence 0 126 58 2 2 0 11
growth 2 11 382 13 6 0 0
flowers 0 0 23 120 7 0 0
senescing 0 2 13 33 116 3 1
senesced 0 3 0 2 29 63 6
no_crop 5 14 8 0 7 25 240
Potential V2 model where I add some shared dense layers
#-------------------------
# The keras model
#-------------------------
input_layer = Input(shape=target_size + (3,))
input_layer = Rescaling(scale = 1./127.5, offset=-1)(input_layer)
base_model = keras.applications.VGG16(
weights=None, # Load weights pre-trained on ImageNet.
#input_shape= target_size + (3,),
input_tensor = input_layer,
include_top=False,
) # Do not include the ImageNet classifier at the top.
base_model_out = base_model.output
base_model_out = keras.layers.Flatten(name='main_flatten')(base_model_out)
base_model_out = keras.layers.Dense(4096, activation='relu', name='shared_dense1')(base_model_out)
base_model_out = keras.layers.Dropout(0.5, name='shared_dropout1')(base_model_out)
base_model_out = keras.layers.Dense(4096, activation='relu', name='shared_dense2')(base_model_out)
base_model_out = keras.layers.Dropout(0.5, name='shared_dropout2')(base_model_out)
def build_category_model(prior_step, class_n, name):
x = keras.layers.Dense(512, activation = 'relu', name=name+'_dense1')(prior_step)
#x = keras.layers.Dropout(0.5, name=name+'_dropout1')(x)
x = keras.layers.Dense(512, activation = 'relu', name=name+'_dense2')(x)
#x = keras.layers.Dropout(0.5, name=name+'_dropout2')(x)
x = keras.layers.Dense(class_n, activation = 'softmax', name=name)(x)
return x
sub_models = [build_category_model(base_model_out, class_n, name) for name, class_n in output_classes.items()]
full_model = keras.Model(base_model.input, sub_models)
Didn't really work out. After 70 epochs of lr 0.01, eps 0.1 Dom cover was worse in all aspects so I didn't bother checking the others.
precision recall f1-score support
unknown 0.31 0.33 0.32 12
vegetation 0.85 0.95 0.89 773
residue 0.77 0.53 0.63 318
soil 0.61 0.53 0.57 135
snow 0.91 0.79 0.85 80
water 0.34 0.88 0.49 24
accuracy 0.79 1342
macro avg 0.63 0.67 0.62 1342
weighted avg 0.80 0.79 0.78 1342
Slightly different implementation with the same result
#-------------------------
# The keras model
#-------------------------
input_layer = Input(shape=target_size + (3,))
input_layer = Rescaling(scale = 1./127.5, offset=-1)(input_layer)
base_model = keras.applications.VGG16(
weights=None, # Load weights pre-trained on ImageNet.
#input_shape= target_size + (3,),
input_tensor = input_layer,
include_top=False,
) # Do not include the ImageNet classifier at the top.
base_model_out = base_model.output
base_model_out = keras.layers.Flatten(name='main_flatten')(base_model_out)
base_model_out = keras.layers.Dense(4096, activation='relu', name='shared_dense1')(base_model_out)
base_model_out = keras.layers.Dense(4096, activation='relu', name='shared_dense2')(base_model_out)
def build_category_model(prior_step, class_n, name):
x = keras.layers.Dense(4096, activation = 'relu', name=name+'_dense1')(prior_step)
x = keras.layers.Dropout(0.5, name=name+'_dropout1')(x)
x = keras.layers.Dense(4096, activation = 'relu', name=name+'_dense2')(x)
x = keras.layers.Dropout(0.5, name=name+'_dropout2')(x)
x = keras.layers.Dense(class_n, activation = 'softmax', name=name)(x)
return x
sub_models = [build_category_model(base_model_out, class_n, name) for name, class_n in output_classes.items()]
full_model = keras.Model(base_model.input, sub_models)
using the "extra" images.
8270 total annotated images 85595 "extra" images 93865 images total
left out sites: arsmorris2, mandani2, cafboydnorthltar01 (10212 images) randomly selected more to reach 20% (18773 total) of all images
fit with learning rate=0.01, epsilon=0.1 for 20 epochs, then lr = 0.001 epsilon=0.1 for 5 epochs.
final file: data/vgg16_v3_25epochs.h5
precision recall f1-score support
unknown 0.24 0.39 0.30 94
vegetation 0.86 0.97 0.91 10998
residue 0.82 0.60 0.69 4403
soil 0.69 0.58 0.63 2054
snow 0.90 0.83 0.87 955
water 0.77 0.95 0.85 269
accuracy 0.83 18773
macro avg 0.71 0.72 0.71 18773
weighted avg 0.83 0.83 0.82 18773
class_description unknown vegetation residue soil snow water
class_description
unknown 37 15 13 9 19 1
vegetation 10 10713 177 60 25 13
residue 3 1245 2636 461 37 21
soil 4 445 378 1194 6 27
snow 100 32 4 10 796 13
water 0 2 12 0 0 255
precision recall f1-score support
uknown 0.31 0.41 0.36 94
unknown_plant 0.87 0.74 0.80 4987
large_grass 0.80 0.95 0.87 2657
small_grass 0.84 0.61 0.71 4116
other 0.51 0.87 0.64 2605
fallow 0.68 0.99 0.81 127
none 0.92 0.81 0.86 4187
accuracy 0.77 18773
macro avg 0.71 0.77 0.72 18773
weighted avg 0.81 0.77 0.78 18773
class_description uknown unknown_plant large_grass small_grass other fallow none
class_description
uknown 39 4 0 4 2 0 45
unknown_plant 3 3712 352 196 506 37 181
large_grass 0 49 2511 16 76 0 5
small_grass 4 213 36 2503 1304 6 50
other 2 101 156 75 2255 1 15
fallow 0 1 0 0 0 126 0
none 76 190 89 176 248 14 3394
precision recall f1-score support
unknown 0.31 0.38 0.34 94
emergence 0.82 0.53 0.64 2924
growth 0.77 0.96 0.85 5968
flowers 0.76 0.78 0.77 1982
senescing 0.62 0.69 0.66 2091
senesced 0.72 0.64 0.68 1521
no_crop 0.93 0.81 0.87 4193
accuracy 0.78 18773
macro avg 0.70 0.68 0.69 18773
weighted avg 0.79 0.78 0.77 18773
class_description unknown emergence growth flowers senescing senesced no_crop
class_description
unknown 36 0 3 0 5 1 49
emergence 0 1545 1150 0 89 9 131
growth 7 119 5702 84 47 2 7
flowers 0 0 344 1537 99 1 1
senescing 0 4 145 389 1441 87 25
senesced 0 2 12 14 474 973 46
no_crop 72 217 73 1 153 277 3400
same as V3 but with the adjusted crop type class which now has corn/soybean/alfalfa/etc explicitly.
8270 original annotated images 255 annotated images dropped due to low "unique category" prevalence.
8015 annotated, usable, mid-day images 83469 "extra" images 91484 images total
18296 validation images (20%) 100k train images resampled from 73188 (91484 - 18296)
left out sites: arsmorris2, mandani2, cafboydnorthltar01 (10212 images) randomly selected more to reach 20% (18773 total) of all images
fit with learning rate=0.01, epsilon=0.1 for 15 epochs, then lr = 0.001 epsilon=0.1 for 5 epochs.
final file: data/vgg16_v4_20epochs.h5
fitting scrip: https://github.com/sdtaylor/PhenocamCNN2/blob/8def77af599bb4b2a139fdfa363a7b6c8c55662a/test_cnn_basic.py
precision recall f1-score support
blurry 0.48 0.46 0.47 96
vegetation 0.90 0.98 0.94 10871
residue 0.80 0.66 0.72 4279
soil 0.63 0.55 0.59 1957
snow 0.90 0.88 0.89 918
water 0.69 0.90 0.78 175
accuracy 0.85 18296
macro avg 0.73 0.74 0.73 18296
weighted avg 0.84 0.85 0.84 18296
class_description blurry vegetation residue soil snow water
class_description
blurry 44 10 22 5 15 0
vegetation 9 10644 153 57 6 2
residue 1 817 2828 571 49 13
soil 2 335 525 1076 15 4
snow 36 13 6 6 806 51
water 0 0 14 3 0 158
precision recall f1-score support
blurry 0.52 0.46 0.49 96
unknown_plant 0.91 0.89 0.90 5623
corn 0.84 0.96 0.90 2413
wheat/barley 0.86 0.60 0.70 2855
soybean 0.37 0.97 0.53 873
alfalfa 0.74 0.96 0.83 170
other 0.91 0.62 0.74 2169
no_crop 0.91 0.88 0.90 4097
accuracy 0.82 18296
macro avg 0.76 0.79 0.75 18296
weighted avg 0.86 0.82 0.83 18296
class_description blurry unknown_plant corn wheat/barley soybean alfalfa other no_crop
class_description
blurry 44 0 0 2 1 1 4 44
unknown_plant 5 4981 196 80 89 43 53 176
corn 2 78 2317 0 9 0 5 2
wheat/barley 0 130 20 1700 834 5 46 120
soybean 0 8 12 0 849 0 0 4
alfalfa 0 1 0 2 1 164 2 0
other 1 36 126 184 466 4 1343 9
no_crop 33 253 88 17 73 6 15 3612
precision recall f1-score support
blurry 0.51 0.46 0.48 96
emergence 0.86 0.71 0.78 2914
growth 0.82 0.95 0.88 5778
flowers 0.83 0.80 0.81 1881
senescing 0.69 0.70 0.69 2054
senesced 0.81 0.67 0.73 1476
no_crop 0.91 0.88 0.90 4097
accuracy 0.83 18296
macro avg 0.77 0.74 0.75 18296
weighted avg 0.83 0.83 0.83 18296
class_description blurry emergence growth flowers senescing senesced no_crop
class_description
blurry 44 0 1 0 5 1 45
emergence 0 2070 554 0 120 1 169
growth 7 189 5483 33 49 3 14
flowers 1 0 291 1506 81 0 2
senescing 0 1 265 276 1439 70 3
senesced 0 1 80 6 293 989 107
no_crop 35 153 35 1 101 164 3608
Vgg16 model using 6714 annotated images. 20k train image sample size, 0.2 validation fraction, (224,224) image size.
lr = 0.01, epsilon = 0.1, 50 epochs
in confusion matrices the columns are predicted and rows are true labels
dominant cover
crop type
crop status