search

luodhhh / ModernTCN

This is an official implementation of "ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis" (ICLR 2024 Spotlight), https://openreview.net/forum?id=vpJMJerXHU
MIT License
100 stars 12 forks source link

Bug Issue: drop_last=True for test dataset may lead to inaccurate test results #1

Open JingzheShi opened 3 months ago

JingzheShi commented 3 months ago

Dear authors:

I am impressed by this amazing work! But when I am reading another paper in ICLR 24' about time series forecasting (FITS: Modeling Time Series with $10k$ Parameters) I noticed the authors reported this bug in their repo (https://github.com/vewoxic/fits).

In short, this bug, originitated from Informer (AAAI 21'), drops some test samples in the last test batch thus resulting in inaccurate results measured. (Please refer to FITS' repo for a detailed description.) Could you please confirm whether your results are affected by this piece of code? If so, could you please modify it and correct the results in the tables in your assey?

Thanks!

ghbty2023 commented 2 months ago

The benchmark paper of PVLDB2024 "TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods" also pointed out that the droplast operation should be abandoned during the testing process, otherwise it will cause unfair comparisons.