Temporal tests - 30D to 10D Presto performance comparison on crop type task

[giollimirgia]

Excluded CROPTYPE_LABEL :

[ 0, 991, 7900, 9900, 9998, # unspecified cropland 
 1910, 1900, 1920, 1000, # cereals
 11, 9910, 6212  # tmp crops ]

Test year: 2021

Training, validation and test distributions

Experiments:

• [x] pretrained Presto (not fine tuned) + Catboost training on crop-type monthly data
• [x] pretrained Presto (not fine tuned) + Catboost training on crop-type decadal data
• [x] Presto FT on (crop/no-crop) monthly data + Catboost training on crop-type monthly data
• [x] Presto FT on (crop/no-crop) decadal data + Catboost training on crop-type decadal data
• [x] Self-Supervised trained Presto on monthly data (#64 ) + Catboost training on crop-type monthly data
• [x] Self-Supervised trained Presto on decadal data (#64) + Catboost training on crop-type decadal data

[giollimirgia]

Pretrained Presto (not fine tuned) + Catboost training on crop-type monthly data

              precision    recall  f1-score   support

   110101000       0.57      0.43      0.49     12801
   110102000       0.50      0.47      0.49     10744
   110103000       0.11      0.37      0.16       731
   110106000       0.63      0.78      0.70      6790
   110107000       0.21      0.54      0.31       592
   110600001       0.93      0.40      0.56     13916
   110600002       0.07      0.43      0.12       235
   110600003       0.96      0.68      0.80     14296
   119999999       0.29      0.67      0.41      8058

    accuracy                           0.55     68163
   macro avg       0.48      0.53      0.45     68163
weighted avg       0.68      0.55      0.57     68163

 f1_score: 0.4479174298321852

[giollimirgia]

Presto FT on (crop/no-crop) monthly data + Catboost training on crop-type monthly data

              precision    recall  f1-score   support

   110101000       0.48      0.56      0.52     12801
   110102000       0.56      0.30      0.39     10744
   110103000       0.10      0.30      0.15       731
   110106000       0.48      0.78      0.60      6790
   110107000       0.06      0.00      0.00       592
   110600001       0.87      0.25      0.38     13916
   110600002       0.18      0.46      0.26       235
   110600003       0.99      0.64      0.78     14296
   119999999       0.27      0.69      0.39      8058

    accuracy                           0.50     68163
   macro avg       0.45      0.44      0.39     68163
weighted avg       0.65      0.50      0.51     68163

 f1_score: 0.3856634547348625

[giollimirgia]

Presto FT on (crop/no-crop) decadal data + Catboost training on crop-type decadal data

              precision    recall  f1-score   support

   110101000       0.44      0.47      0.46     12801
   110102000       0.51      0.27      0.35     10744
   110103000       0.18      0.22      0.20       731
   110106000       0.30      0.82      0.44      6790
   110107000       0.34      0.15      0.21       592
   110600001       0.90      0.15      0.25     13916
   110600002       0.24      0.13      0.17       235
   110600003       0.99      0.56      0.72     14296
   119999999       0.27      0.64      0.38      8058

    accuracy                           0.44     68163
   macro avg       0.47      0.38      0.35     68163
weighted avg       0.62      0.44      0.44     68163

 f1_score: 0.3540460669977847

[giollimirgia]

Pretrained Presto (not fine tuned) + Catboost training on crop-type decadal data

              precision    recall  f1-score   support

   110101000       0.48      0.38      0.42     12801
   110102000       0.48      0.29      0.36     10744
   110103000       0.05      0.38      0.10       731
   110106000       0.46      0.70      0.55      6790
   110107000       0.24      0.76      0.37       592
   110600001       0.82      0.12      0.22     13916
   110600002       0.08      0.09      0.09       235
   110600003       0.93      0.57      0.71     14296
   119999999       0.21      0.59      0.30      8058

    accuracy                           0.41     68163
   macro avg       0.42      0.43      0.35     68163
weighted avg       0.60      0.41      0.42     68163

 f1_score: 0.3466335894838581

[giollimirgia]

Self-Supervised trained Presto on monthly data + Catboost training on crop-type monthly data

              precision    recall  f1-score   support

   110101000       0.50      0.61      0.55     12801
   110102000       0.53      0.27      0.36     10744
   110103000       0.10      0.38      0.16       731
   110106000       0.60      0.80      0.68      6790
   110107000       0.21      0.32      0.26       592
   110600001       0.93      0.41      0.57     13916
   110600002       0.08      0.38      0.13       235
   110600003       0.92      0.67      0.77     14296
   119999999       0.29      0.59      0.39      8058

    accuracy                           0.54     68163
   macro avg       0.46      0.49      0.43     68163
weighted avg       0.66      0.54      0.56     68163

 f1_score: 0.4298634163063877

[giollimirgia]

Self-Supervised trained Presto on decadal data + Catboost training on crop-type decadal data

              precision    recall  f1-score   support

   110101000       0.53      0.57      0.55     12801
   110102000       0.49      0.21      0.29     10744
   110103000       0.08      0.28      0.12       731
   110106000       0.59      0.61      0.60      6790
   110107000       0.25      0.86      0.39       592
   110600001       0.88      0.19      0.32     13916
   110600002       1.00      0.01      0.02       235
   110600003       0.87      0.75      0.80     14296
   119999999       0.19      0.55      0.28      8058

    accuracy                           0.47     68163
   macro avg       0.54      0.45      0.38     68163
weighted avg       0.63      0.47      0.48     68163

 f1_score: 0.375170481887316

[giollimirgia]

Experiment F1 score summary

[gabrieltseng]

Closing - summary in #63