search

yurujaja / pangaea-bench

Towards Robust Evaluation for Geospatial Foundation Models
GNU General Public License v3.0
102 stars 10 forks source link

Best checkpoint metric #87

Closed SebastianHafner closed 1 month ago

SebastianHafner commented 1 month ago

The best checkpoint metric has now been added to the task configs.

If the metric is a list of per-class values, we use the value of the foreground class when the number of classes is 1 and the average of the per-class values otherwise (as discussed).

The only awkward thing is retrieving the number of classes from the dataset. I had to include a separate clause for the limited-label scenario since the instance variable 'dataset' changes from a 'Dataset' object to a 'Subset' object when we are not training on the full dataset.

gle-bellier commented 1 month ago

Hi Sebastian, For the specific case where the dataset is a Subset, it was also an issue discussed in #82. Once the changes of #82 are merged to the main branch we can remove the if-else conditioning on the dataset type.