search

sdtaylor / PhenocamCNN2

Other
1 stars 0 forks source link

multi-label model classification notes #3

Open sdtaylor opened 3 years ago

sdtaylor commented 3 years ago

Vgg16 model using 6714 annotated images. 20k train image sample size, 0.2 validation fraction, (224,224) image size.

lr = 0.01, epsilon = 0.1, 50 epochs

in confusion matrices the columns are predicted and rows are true labels

dominant cover

              precision    recall  f1-score   support

     unknown       0.59      0.71      0.65        14
  vegetation       0.90      0.91      0.91       793
     residue       0.67      0.65      0.66       293
        soil       0.57      0.53      0.55       144
        snow       0.90      0.77      0.83        82
       water       0.41      0.81      0.54        16

    accuracy                           0.80      1342
   macro avg       0.67      0.73      0.69      1342
weighted avg       0.81      0.80      0.80      1342

class_description  unknown  vegetation  residue  soil  snow  water
class_description                                                 
unknown                 10           1        3     0     0      0
vegetation               0         725       40    19     5      4
residue                  1          61      190    33     2      6
soil                     0          15       45    77     0      7
snow                     6           1        5     5    63      2
water                    0           0        2     1     0     13

crop type

               precision    recall  f1-score   support

       uknown       0.56      0.71      0.63        14
unknown_plant       0.74      0.63      0.68       327
  large_grass       0.67      0.94      0.78       238
  small_grass       0.78      0.80      0.79       258
        other       0.67      0.54      0.60       188
       fallow       0.38      0.87      0.53        15
         none       0.86      0.75      0.80       302

     accuracy                           0.73      1342
    macro avg       0.67      0.75      0.69      1342
 weighted avg       0.75      0.73      0.73      1342

class_description  uknown  unknown_plant  large_grass  small_grass  other  fallow  none
class_description                                                                      
uknown                 10              1            0            0      0       0     3
unknown_plant           0            205           48           15     22       7    30
large_grass             0              9          223            4      2       0     0
small_grass             0             12           16          206     21       2     1
other                   0             18           38           27    101       1     3
fallow                  0              0            0            0      1      13     1
none                    8             33            7           12      3      11   228

crop status

             precision    recall  f1-score   support

     unknown       0.56      0.71      0.63        14
   emergence       0.70      0.58      0.63       190
      growth       0.80      0.91      0.85       438
     flowers       0.76      0.69      0.72       160
   senescing       0.58      0.64      0.60       138
    senesced       0.56      0.55      0.55       100
     no_crop       0.87      0.78      0.83       302

    accuracy                           0.75      1342
   macro avg       0.69      0.70      0.69      1342
weighted avg       0.75      0.75      0.75      1342

class_description  unknown  emergence  growth  flowers  senescing  senesced  no_crop
class_description                                                                   
unknown                 10          0       1        0          0         0        3
emergence                0        110      53        1          4         2       20
growth                   0         22     400        8          5         3        0
flowers                  0          0      26      110         23         1        0
senescing                0          0      11       25         88        12        2
senesced                 0          5       2        1         28        55        9
no_crop                  8         20       6        0          5        26      237
sdtaylor commented 3 years ago

same as above but going another 20 epochs w/ lr at 0.001

dominant cover

              precision    recall  f1-score   support

     unknown       0.67      0.71      0.69        14
  vegetation       0.89      0.95      0.92       793
     residue       0.77      0.65      0.70       293
        soil       0.67      0.64      0.65       144
        snow       0.94      0.73      0.82        82
       water       0.38      0.88      0.53        16

    accuracy                           0.83      1342
   macro avg       0.72      0.76      0.72      1342
weighted avg       0.84      0.83      0.83      1342

class_description  unknown  vegetation  residue  soil  snow  water
class_description                                                 
unknown                 10           2        1     0     0      1
vegetation               0         752       23    13     2      3
residue                  1          62      190    32     2      6
soil                     0          24       25    92     0      3
snow                     4           1        6     1    60     10
water                    0           0        2     0     0     14

crop type

           precision    recall  f1-score   support

       uknown       0.67      0.71      0.69        14
unknown_plant       0.80      0.61      0.70       327
  large_grass       0.72      0.93      0.81       238
  small_grass       0.83      0.67      0.74       258
        other       0.53      0.72      0.61       188
       fallow       0.68      0.87      0.76        15
         none       0.86      0.81      0.83       302

     accuracy                           0.74      1342
    macro avg       0.73      0.76      0.74      1342
 weighted avg       0.77      0.74      0.75      1342

 class_description  uknown  unknown_plant  large_grass  small_grass  other  fallow  none
class_description                                                                      
uknown                 10              1            0            0      0       0     3
unknown_plant           0            201           36           10     47       4    29
large_grass             0              7          222            3      6       0     0
small_grass             0              8           20          173     54       0     3
other                   0              8           28           12    136       0     4
fallow                  0              0            0            0      1      13     1
none                    5             25            4           10     12       2   244

crop status

              precision    recall  f1-score   support

     unknown       0.67      0.71      0.69        14
   emergence       0.71      0.54      0.61       190
      growth       0.80      0.89      0.84       438
     flowers       0.74      0.86      0.80       160
   senescing       0.61      0.62      0.62       138
    senesced       0.66      0.63      0.65       100
     no_crop       0.89      0.81      0.85       302

    accuracy                           0.77      1342
   macro avg       0.73      0.72      0.72      1342
weighted avg       0.77      0.77      0.77      1342

class_description  unknown  emergence  growth  flowers  senescing  senesced  no_crop
class_description                                                                   
unknown                 10          0       1        0          0         0        3
emergence                0        102      62        1          3         3       19
growth                   0         20     391       19          7         0        1
flowers                  0          0      12      137         11         0        0
senescing                0          0      13       26         86        12        1
senesced                 0          2       2        1         25        63        7
no_crop                  5         20       8        0          8        17      244
sdtaylor commented 3 years ago

starting again from 55 epochs and only going 5 more with lr 0.001

this version: https://github.com/sdtaylor/PhenocamCNN2/blob/15594755e01b60ffb2b13521865a878c7fbb51f1/test_cnn_basic.py

this model saved in vgg16_v1_70epochs.h5

dominant cover

              precision    recall  f1-score   support

     unknown       0.40      0.44      0.42         9
  vegetation       0.90      0.94      0.92       776
     residue       0.75      0.72      0.73       296
        soil       0.72      0.62      0.66       166
        snow       0.95      0.78      0.85        72
       water       0.53      0.78      0.63        23

    accuracy                           0.84      1342
   macro avg       0.71      0.71      0.70      1342
weighted avg       0.84      0.84      0.84      1342

class_description  unknown  vegetation  residue  soil  snow  water
class_description                                                 
unknown                  4           3        0     0     2      0
vegetation               1         732       29    12     0      2
residue                  0          51      212    29     1      3
soil                     0          23       36   103     0      4
snow                     4           1        4     0    56      7
water                    1           2        2     0     0     18

crop type

               precision    recall  f1-score   support

       uknown       0.44      0.44      0.44         9
unknown_plant       0.85      0.76      0.80       331
  large_grass       0.80      0.94      0.86       259
  small_grass       0.78      0.66      0.71       235
        other       0.61      0.73      0.66       197
       fallow       0.52      0.92      0.67        12
         none       0.91      0.84      0.88       299

     accuracy                           0.79      1342
    macro avg       0.70      0.76      0.72      1342
 weighted avg       0.80      0.79      0.79      1342

class_description  uknown  unknown_plant  large_grass  small_grass  other  fallow  none
class_description                                                                      
uknown                  4              1            0            2      0       0     2
unknown_plant           0            251           25           13     24       4    14
large_grass             0              9          244            3      3       0     0
small_grass             1             10           11          154     54       0     5
other                   0              9           20           22    143       0     3
fallow                  0              0            0            0      0      11     1
none                    4             17            6            4     10       6   252

crop status

              precision    recall  f1-score   support

     unknown       0.36      0.44      0.40         9
   emergence       0.81      0.63      0.71       199
      growth       0.79      0.92      0.85       414
     flowers       0.70      0.80      0.75       150
   senescing       0.69      0.69      0.69       168
    senesced       0.69      0.61      0.65       103
     no_crop       0.92      0.80      0.86       299

    accuracy                           0.78      1342
   macro avg       0.71      0.70      0.70      1342
weighted avg       0.79      0.78      0.78      1342

class_description  unknown  emergence  growth  flowers  senescing  senesced  no_crop
class_description                                                                   
unknown                  4          0       1        1          1         0        2
emergence                0        126      58        2          2         0       11
growth                   2         11     382       13          6         0        0
flowers                  0          0      23      120          7         0        0
senescing                0          2      13       33        116         3        1
senesced                 0          3       0        2         29        63        6
no_crop                  5         14       8        0          7        25      240
sdtaylor commented 3 years ago

Potential V2 model where I add some shared dense layers

#-------------------------
# The keras model
#-------------------------
input_layer = Input(shape=target_size + (3,))
input_layer =  Rescaling(scale = 1./127.5, offset=-1)(input_layer)

base_model = keras.applications.VGG16(
    weights=None,  # Load weights pre-trained on ImageNet.
    #input_shape= target_size + (3,),
    input_tensor = input_layer,
    include_top=False,
)  # Do not include the ImageNet classifier at the top.
base_model_out = base_model.output

base_model_out = keras.layers.Flatten(name='main_flatten')(base_model_out)
base_model_out = keras.layers.Dense(4096, activation='relu', name='shared_dense1')(base_model_out)
base_model_out = keras.layers.Dropout(0.5, name='shared_dropout1')(base_model_out)
base_model_out = keras.layers.Dense(4096, activation='relu', name='shared_dense2')(base_model_out)
base_model_out = keras.layers.Dropout(0.5, name='shared_dropout2')(base_model_out)

def build_category_model(prior_step, class_n, name):
    x = keras.layers.Dense(512, activation = 'relu', name=name+'_dense1')(prior_step)
    #x = keras.layers.Dropout(0.5, name=name+'_dropout1')(x)
    x = keras.layers.Dense(512, activation = 'relu', name=name+'_dense2')(x)
    #x = keras.layers.Dropout(0.5, name=name+'_dropout2')(x)
    x = keras.layers.Dense(class_n,  activation = 'softmax', name=name)(x)
    return x

sub_models = [build_category_model(base_model_out, class_n, name) for name, class_n in output_classes.items()]

full_model = keras.Model(base_model.input, sub_models)

Didn't really work out. After 70 epochs of lr 0.01, eps 0.1 Dom cover was worse in all aspects so I didn't bother checking the others. 

           precision    recall  f1-score   support

     unknown       0.31      0.33      0.32        12
  vegetation       0.85      0.95      0.89       773
     residue       0.77      0.53      0.63       318
        soil       0.61      0.53      0.57       135
        snow       0.91      0.79      0.85        80
       water       0.34      0.88      0.49        24

    accuracy                           0.79      1342
   macro avg       0.63      0.67      0.62      1342
weighted avg       0.80      0.79      0.78      1342

Slightly different implementation with the same result

#-------------------------
# The keras model
#-------------------------
input_layer = Input(shape=target_size + (3,))
input_layer =  Rescaling(scale = 1./127.5, offset=-1)(input_layer)

base_model = keras.applications.VGG16(
    weights=None,  # Load weights pre-trained on ImageNet.
    #input_shape= target_size + (3,),
    input_tensor = input_layer,
    include_top=False,
)  # Do not include the ImageNet classifier at the top.
base_model_out = base_model.output

base_model_out = keras.layers.Flatten(name='main_flatten')(base_model_out)
base_model_out = keras.layers.Dense(4096, activation='relu', name='shared_dense1')(base_model_out)
base_model_out = keras.layers.Dense(4096, activation='relu', name='shared_dense2')(base_model_out)

def build_category_model(prior_step, class_n, name):
    x = keras.layers.Dense(4096, activation = 'relu', name=name+'_dense1')(prior_step)
    x = keras.layers.Dropout(0.5, name=name+'_dropout1')(x)
    x = keras.layers.Dense(4096, activation = 'relu', name=name+'_dense2')(x)
    x = keras.layers.Dropout(0.5, name=name+'_dropout2')(x)
    x = keras.layers.Dense(class_n,  activation = 'softmax', name=name)(x)
    return x

sub_models = [build_category_model(base_model_out, class_n, name) for name, class_n in output_classes.items()]

full_model = keras.Model(base_model.input, sub_models)
sdtaylor commented 3 years ago

version 3 classification results

using the "extra" images.

8270 total annotated images 85595 "extra" images 93865 images total

left out sites: arsmorris2, mandani2, cafboydnorthltar01 (10212 images) randomly selected more to reach 20% (18773 total) of all images

fit with learning rate=0.01, epsilon=0.1 for 20 epochs, then lr = 0.001 epsilon=0.1 for 5 epochs.

final file: data/vgg16_v3_25epochs.h5

Dominant cover

              precision    recall  f1-score   support

     unknown       0.24      0.39      0.30        94
  vegetation       0.86      0.97      0.91     10998
     residue       0.82      0.60      0.69      4403
        soil       0.69      0.58      0.63      2054
        snow       0.90      0.83      0.87       955
       water       0.77      0.95      0.85       269

    accuracy                           0.83     18773
   macro avg       0.71      0.72      0.71     18773
weighted avg       0.83      0.83      0.82     18773

class_description  unknown  vegetation  residue  soil  snow  water
class_description                                                 
unknown                 37          15       13     9    19      1
vegetation              10       10713      177    60    25     13
residue                  3        1245     2636   461    37     21
soil                     4         445      378  1194     6     27
snow                   100          32        4    10   796     13
water                    0           2       12     0     0    255

Crop type

               precision    recall  f1-score   support

       uknown       0.31      0.41      0.36        94
unknown_plant       0.87      0.74      0.80      4987
  large_grass       0.80      0.95      0.87      2657
  small_grass       0.84      0.61      0.71      4116
        other       0.51      0.87      0.64      2605
       fallow       0.68      0.99      0.81       127
         none       0.92      0.81      0.86      4187

     accuracy                           0.77     18773
    macro avg       0.71      0.77      0.72     18773
 weighted avg       0.81      0.77      0.78     18773

class_description  uknown  unknown_plant  large_grass  small_grass  other  fallow  none
class_description                                                                      
uknown                 39              4            0            4      2       0    45
unknown_plant           3           3712          352          196    506      37   181
large_grass             0             49         2511           16     76       0     5
small_grass             4            213           36         2503   1304       6    50
other                   2            101          156           75   2255       1    15
fallow                  0              1            0            0      0     126     0
none                   76            190           89          176    248      14  3394

Crop status

              precision    recall  f1-score   support

     unknown       0.31      0.38      0.34        94
   emergence       0.82      0.53      0.64      2924
      growth       0.77      0.96      0.85      5968
     flowers       0.76      0.78      0.77      1982
   senescing       0.62      0.69      0.66      2091
    senesced       0.72      0.64      0.68      1521
     no_crop       0.93      0.81      0.87      4193

    accuracy                           0.78     18773
   macro avg       0.70      0.68      0.69     18773
weighted avg       0.79      0.78      0.77     18773

class_description  unknown  emergence  growth  flowers  senescing  senesced  no_crop
class_description                                                                   
unknown                 36          0       3        0          5         1       49
emergence                0       1545    1150        0         89         9      131
growth                   7        119    5702       84         47         2        7
flowers                  0          0     344     1537         99         1        1
senescing                0          4     145      389       1441        87       25
senesced                 0          2      12       14        474       973       46
no_crop                 72        217      73        1        153       277     3400
sdtaylor commented 3 years ago

version 4 classification results

same as V3 but with the adjusted crop type class which now has corn/soybean/alfalfa/etc explicitly.

8270 original annotated images 255 annotated images dropped due to low "unique category" prevalence.

8015 annotated, usable, mid-day images 83469 "extra" images 91484 images total

18296 validation images (20%) 100k train images resampled from 73188 (91484 - 18296)

left out sites: arsmorris2, mandani2, cafboydnorthltar01 (10212 images) randomly selected more to reach 20% (18773 total) of all images

fit with learning rate=0.01, epsilon=0.1 for 15 epochs, then lr = 0.001 epsilon=0.1 for 5 epochs.

final file: data/vgg16_v4_20epochs.h5

fitting scrip: https://github.com/sdtaylor/PhenocamCNN2/blob/8def77af599bb4b2a139fdfa363a7b6c8c55662a/test_cnn_basic.py

Dominant cover

           precision    recall  f1-score   support

      blurry       0.48      0.46      0.47        96
  vegetation       0.90      0.98      0.94     10871
     residue       0.80      0.66      0.72      4279
        soil       0.63      0.55      0.59      1957
        snow       0.90      0.88      0.89       918
       water       0.69      0.90      0.78       175

    accuracy                           0.85     18296
   macro avg       0.73      0.74      0.73     18296
weighted avg       0.84      0.85      0.84     18296

class_description  blurry  vegetation  residue  soil  snow  water
class_description                                                
blurry                 44          10       22     5    15      0
vegetation              9       10644      153    57     6      2
residue                 1         817     2828   571    49     13
soil                    2         335      525  1076    15      4
snow                   36          13        6     6   806     51
water                   0           0       14     3     0    158

Crop type

             precision    recall  f1-score   support

       blurry       0.52      0.46      0.49        96
unknown_plant       0.91      0.89      0.90      5623
         corn       0.84      0.96      0.90      2413
 wheat/barley       0.86      0.60      0.70      2855
      soybean       0.37      0.97      0.53       873
      alfalfa       0.74      0.96      0.83       170
        other       0.91      0.62      0.74      2169
      no_crop       0.91      0.88      0.90      4097

     accuracy                           0.82     18296
    macro avg       0.76      0.79      0.75     18296
 weighted avg       0.86      0.82      0.83     18296

 class_description  blurry  unknown_plant  corn  wheat/barley  soybean  alfalfa  other  no_crop
class_description                                                                             
blurry                 44              0     0             2        1        1      4       44
unknown_plant           5           4981   196            80       89       43     53      176
corn                    2             78  2317             0        9        0      5        2
wheat/barley            0            130    20          1700      834        5     46      120
soybean                 0              8    12             0      849        0      0        4
alfalfa                 0              1     0             2        1      164      2        0
other                   1             36   126           184      466        4   1343        9
no_crop                33            253    88            17       73        6     15     3612

Crop status

              precision    recall  f1-score   support

      blurry       0.51      0.46      0.48        96
   emergence       0.86      0.71      0.78      2914
      growth       0.82      0.95      0.88      5778
     flowers       0.83      0.80      0.81      1881
   senescing       0.69      0.70      0.69      2054
    senesced       0.81      0.67      0.73      1476
     no_crop       0.91      0.88      0.90      4097

    accuracy                           0.83     18296
   macro avg       0.77      0.74      0.75     18296
weighted avg       0.83      0.83      0.83     18296

class_description  blurry  emergence  growth  flowers  senescing  senesced  no_crop
class_description                                                                  
blurry                 44          0       1        0          5         1       45
emergence               0       2070     554        0        120         1      169
growth                  7        189    5483       33         49         3       14
flowers                 1          0     291     1506         81         0        2
senescing               0          1     265      276       1439        70        3
senesced                0          1      80        6        293       989      107
no_crop                35        153      35        1        101       164     3608