Why use drop_last=True in test (and val) dataloader?

thuml / Autoformer

About Code release for "Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting" (NeurIPS 2021), https://arxiv.org/abs/2106.13008

MIT License

2k stars 430 forks source link

Why use drop_last=True in test (and val) dataloader? #104

Closed oguiza closed 1 year ago

oguiza commented 1 year ago

Hi, First, thanks for the excellent paper and for sharing this repo. Great work!

I'd like to ask how is the reported performance calculated. Is it based on the test dataset?
If the answer is yes, why do you set the test dataloader drop_last=True? This would mean that performance is not reported for all samples in the test dataset (some samples will be dropped, which is not what we want). In addition to this, changes in the batch size would lead to reporting performance on a different number of samples.

wuhaixu2016 commented 1 year ago

Hi, thanks for your insightful questions.

Yes, the results are calculated from the test dataset.
Thanks for pointing out this. I think you are right. We have reevaluated all the results by setting drop_last=False. And the final results do not change. This may come from that the dropped samples are a small number of the whole test dataset. We have fixed this in the latest commit: https://github.com/thuml/Autoformer/commit/d9100709b04e3e8361170794eba4f47b1afb217f By the way, during the evaluation, I think the best way is to set the batch size as 1, which is just matching the online serving scenario.

oguiza commented 1 year ago

Wow, thanks a lot for your response! I found the issue because I was using different batch sizes. It’s good to have a model that measures performance independently of batch size. Thanks again for the great work you did with Autoformer!