search

thuml / Time-Series-Library

A Library for Advanced Deep Time Series Models.
MIT License
6.62k stars 1.04k forks source link

The problem of reproduction of model results #303

Closed Jackson906E closed 10 months ago

Jackson906E commented 10 months ago

I selected TimeNet and found that the result of ETTm2 dataset (prediction length=96) was (mse:0.124, mae:0.236) while the result in the paper was (mse:0.187, mae:0.267). I want to know why the result will be so much difference? The random seed I use is also source code to give.

wuhaixu2016 commented 10 months ago

Hi, according to your question, the reproduced results are better than our paper.

I think you may recheck the training process. If possible, would you please provide the training log?

Jackson906E commented 10 months ago

When I ran TimesNet on the ETTm1 dataset, the results were even worse than that in the paper. (Each metric is 0.1 higher than the results in the paper. e.g. for prediction length=96, (mse:0.460, mae:0.450), while the result in the paper (mse:0.338, mae:0.375) I was really confused by the results I reproduced. TimesNet_ETTm1_96_96.log I will appreciate for your clarification.

Jackson906E commented 10 months ago

Hi, Haixu

When I ran TimesNet on the ETTm1 dataset, the results were even worse than that in the paper. (Each metric is 0.1 higher than the results in the paper. e.g. for prediction length=96, (mse:0.460, mae:0.450), while the result in the paper (mse:0.338, mae:0.375) I was really confused by the results I reproduced. TimesNet_ETTm1_96_96.loghttps://github.com/thuml/Time-Series-Library/files/13650733/TimesNet_ETTm1_96_96.log I will appreciate for your clarification.

Best Regards

发件人: wuhaixu2016 @.> 日期: 星期二, 2023年12月12日 21:44 收件人: thuml/Time-Series-Library @.> 抄送: Xiang LI @.>, Author @.> 主题: Re: [thuml/Time-Series-Library] The problem of reproduction of model results (Issue #303) You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

Hi, according to your question, the reproduced results are better than our paper.

I think you may recheck the training process. If possible, would you please provide the training log?

― Reply to this email directly, view it on GitHubhttps://github.com/thuml/Time-Series-Library/issues/303#issuecomment-1852066052, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2ZI3KLD627JMWEKMF7O6FLYJBNUHAVCNFSM6AAAAABARCDVP2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJSGA3DMMBVGI. You are receiving this because you authored the thread.Message ID: @.***>

wuhaixu2016 commented 10 months ago

Hi, Thanks for your detailed log.

I found that the printed dataset samples in your log are different from mine. Please make sure that you have used the latest version of TSlib and download the dataset from our provided Google Drive.

And here is my experiment log, which can produce the results well. TimesNet_96_96.log