WorldCereal / presto-worldcereal

2 stars 0 forks source link

Compare monthly to 10D Presto performance on crop/no-crop task #52

Closed kvantricht closed 2 months ago

kvantricht commented 4 months ago

Now that 10D Presto results are finalized, last thing to do is a formal comparison in a controlled way of the downstream Catboost results of crop/no-crop prediction between the monthly data processed by @gabrieltseng and the 10D data processed by @giollimirgia.

Expected outcome is proof whether or not we can justify for WorldCereal Phase II that we can stick to monthly data. For this, the results of both independent simulations need to be brought together in a table comparing one to one the results, verifying if any (significant) differences appear.

kvantricht commented 2 months ago

@giollimirgia once your current experiments are concluded, can you shortly report here and close the issue? If required (e.g. for crop type), we should open a new ticket to track the next steps.

giollimirgia commented 2 months ago

f1_score_comparison

Comparison of F1-scores related to a baseline Catboost model trained on decadal expert features, and two additional Catboost models trained on distinct variations of Presto embeddings. In the latter instances, Presto underwent fine-tuning based on monthly and decadal features, respectively. The equivalent performance achieved in the experiments involving Presto suggests the necessary information for addressing the binary crop/no-crop classification task is embedded in both monthly and decadal features. It can also be assumed that monthly composites are by their very nature more robust through time as they contain less internal variation and are therefore natively more robust throughout multiple years. However, the effect of the compositing window on performance might be more visible in tasks like crop-type classification, where the monthly features might lack the temporal detail to capture highly dynamic nature of crop growth compared to decadal features. Further validation on such multi-class settings should/will be carried out.