Stratified tests - 30D to 10D Presto performance comparison on crop type task

[giollimirgia]

Excluded CROPTYPE_LABEL :

[ 0, 991, 7900, 9900, 9998, # unspecified cropland 
 1910, 1900, 1920, 1000, # cereals
 11, 9910, 6212  # tmp crops ]

Training, validation and test distributions

Experiments:

• [x] pretrained Presto (not fine tuned) + Catboost training on crop-type monthly data
• [x] pretrained Presto (not fine tuned) + Catboost training on crop-type decadal data
• [x] Presto FT on (crop/no-crop) monthly data + Catboost training on crop-type monthly data
• [x] Presto FT on (crop/no-crop) decadal data + Catboost training on crop-type decadal data
• [x] Self-Supervised trained Presto on monthly data (#64) + Catboost training on crop-type monthly data
• [x] Self-Supervised trained Presto on decadal data (#64) + Catboost training on crop-type decadal data

[giollimirgia]

Pretrained Presto (not fine tuned) + Catboost training on crop-type monthly data

              precision    recall  f1-score   support

   110101000       0.78      0.65      0.71      8940
   110102000       0.58      0.65      0.61      5012
   110103000       0.32      0.58      0.41       697
   110106000       0.85      0.76      0.80      8885
   110107000       0.44      0.77      0.56       489
   110600001       0.76      0.89      0.82      3615
   110600002       0.71      0.87      0.78      3069
   110600003       0.81      0.91      0.86      3420
   119999999       0.70      0.60      0.65      6424

    accuracy                           0.73     40551
   macro avg       0.66      0.74      0.69     40551
weighted avg       0.74      0.73      0.73     40551

 f1_score: 0.6893964818570302

[giollimirgia]

Presto FT on (crop/no-crop) monthly data + Catboost training on crop-type monthly data

              precision    recall  f1-score   support

   110101000       0.79      0.67      0.73      8940
   110102000       0.58      0.68      0.63      5012
   110103000       0.36      0.56      0.44       697
   110106000       0.87      0.79      0.83      8885
   110107000       0.46      0.74      0.57       489
   110600001       0.84      0.89      0.86      3615
   110600002       0.72      0.88      0.79      3069
   110600003       0.90      0.92      0.91      3420
   119999999       0.69      0.66      0.67      6424

    accuracy                           0.75     40551
   macro avg       0.69      0.75      0.71     40551
weighted avg       0.76      0.75      0.75     40551

 f1_score: 0.7140755058325549

[giollimirgia]

Pretrained Presto (not fine tuned) + Catboost training on crop-type decadal data

              precision    recall  f1-score   support

   110101000       0.77      0.66      0.71      9007
   110102000       0.59      0.65      0.62      5010
   110103000       0.31      0.52      0.39       669
   110106000       0.82      0.74      0.78      8936
   110107000       0.42      0.73      0.54       445
   110600001       0.79      0.89      0.84      3691
   110600002       0.70      0.87      0.77      3112
   110600003       0.78      0.91      0.84      3391
   119999999       0.69      0.60      0.64      6290

    accuracy                           0.72     40551
   macro avg       0.65      0.73      0.68     40551
weighted avg       0.73      0.72      0.72     40551

 f1_score: 0.68080878597037

[giollimirgia]

Presto FT on (crop/no-crop) decadal data + Catboost training on crop-type decadal data

              precision    recall  f1-score   support

   110101000       0.80      0.70      0.75      9007
   110102000       0.62      0.70      0.66      5010
   110103000       0.39      0.54      0.45       669
   110106000       0.85      0.80      0.83      8936
   110107000       0.47      0.72      0.57       445
   110600001       0.85      0.89      0.87      3691
   110600002       0.75      0.87      0.81      3112
   110600003       0.88      0.92      0.90      3391
   119999999       0.70      0.67      0.68      6290

    accuracy                           0.76     40551
   macro avg       0.70      0.76      0.72     40551
weighted avg       0.77      0.76      0.77     40551

 f1_score: 0.7235939948683406

[giollimirgia]

Self-Supervised trained Presto on monthly data + Catboost training on crop-type monthly data

              precision    recall  f1-score   support

   110101000       0.77      0.66      0.71      8940
   110102000       0.57      0.64      0.60      5012
   110103000       0.34      0.57      0.43       697
   110106000       0.84      0.76      0.80      8885
   110107000       0.45      0.76      0.57       489
   110600001       0.79      0.89      0.84      3615
   110600002       0.71      0.88      0.79      3069
   110600003       0.83      0.91      0.87      3420
   119999999       0.71      0.61      0.66      6424

    accuracy                           0.73     40551
   macro avg       0.67      0.74      0.69     40551
weighted avg       0.74      0.73      0.73     40551

 f1_score: 0.694962735300899

[giollimirgia]

Self-Supervised trained Presto on decadal data + Catboost training on crop-type decadal data

              precision    recall  f1-score   support

   110101000       0.77      0.68      0.72      9007
   110102000       0.60      0.66      0.63      5010
   110103000       0.35      0.53      0.42       669
   110106000       0.83      0.76      0.79      8936
   110107000       0.47      0.77      0.59       445
   110600001       0.80      0.90      0.85      3691
   110600002       0.72      0.87      0.79      3112
   110600003       0.85      0.93      0.89      3391
   119999999       0.71      0.63      0.67      6290

    accuracy                           0.74     40551
   macro avg       0.68      0.75      0.70     40551
weighted avg       0.75      0.74      0.74     40551

 f1_score: 0.7037557986429195

[giollimirgia]

Experiment F1 score overview

[gabrieltseng]

Closing - summary in #63