search

amazon-science / earth-forecasting-transformer

Official implementation of Earthformer
Apache License 2.0
350 stars 60 forks source link

Question about the calculation of metrics #3

Closed bugsuse closed 1 year ago

bugsuse commented 1 year ago

It's a very solid and interesting work! Thanks for your sharing the nice code!

I found that the arguments order of function calc_seq_hits_misses_fas at L144 seems inconsistent with the definition at L112. This perhaps lead to errors in the calculation of some metrics (such as pod, bias and sucr).

https://github.com/amazon-research/earth-forecasting-transformer/blob/9c18c5a0be9c1af686827a191217a6cccf57812b/src/earthformer/metrics/sevir.py#L112

https://github.com/amazon-research/earth-forecasting-transformer/blob/9c18c5a0be9c1af686827a191217a6cccf57812b/src/earthformer/metrics/sevir.py#L144

gaozhihan commented 1 year ago

Thanks for your interest and reporting the error! We will submit a PR for fixing it. The incorrect usage of calc_seq_hits_misses_fas at L144 does not affect the CSI reported in our paper, but results in the reported pod to be the actual sucr, the reported sucr to be the actual pod, and the reported bias to be the actual 1/bias. We will also soon submitting PRs that add test-cases, inference examples with the pretrained model.

sxjscience commented 1 year ago

Really appreciate for pointing it out! @bugsuse

bugsuse commented 1 year ago

Happy to help! I have also learned many advanced usages from the nice and clean code you shared! @sxjscience