Results are too sensitive to hyperparameters

[crishna0401]

Hi, I just found out that adjusted learning rate, batch size significantly (increase/decrease) changes the results. Although all of the compared methods are using same style of training, I feel the hyperparameters shouldn't be the deciding factor.

[hanlu-nju]

Hyperparameters like learning rate and batch size are not deciding factors, they are influencing factors. All the deep learning models (Transformer, MLP, CNN) are influenced by the choice of learning rate and batch size. The study of the relationship between batch size and learning rate, and their impact on generalization has been a particular field. If you are interested, we refer to [1] [2] [3] and their related research for further study.

Our model chooses almost the same learning rate (0.0003) and batch size (16 or 32) for all the datasets. We think our model, along with the choice of hyperparameters, is robust across different circumstances.

[crishna0401]

I understand the influence of hyperparameters. But my concern is for some datasets model is converged within 1-4 epochs itself. what is your comment on that.

For example I ran weather data lookback=96 and pred_len=96, the model converged in first epoch itself.

Args in experiment: Namespace(task_name='long_term_forecast', is_training=1, model_id='weather_96_96', model='SOFTS', data='custom', root_path='/DATA/krishna/mvltf/dataset/', data_path='weather.csv', features='M', target='OT', freq='h', checkpoints='./checkpoints/', seq_len=96, label_len=48, pred_len=96, seasonal_patterns='Monthly', enc_in=21, dec_in=21, c_out=21, d_model=512, d_core=128, e_layers=3, d_layers=1, d_ff=512, moving_avg=25, factor=1, distil=True, dropout=0.0, embed='timeF', activation='gelu', output_attention=False, attention_type='full', use_norm=True, num_workers=4, itr=1, train_epochs=10, batch_size=16, patience=10, learning_rate=0.0003, des='Exp', loss='MSE', lradj='cosine', use_amp=False, use_gpu=True, gpu=0, use_multi_gpu=False, devices='0,1,2,3', save_model=False) Use GPU: cuda:0

start training : long_term_forecast_weather_96_96_SOFTS_custom_ftM_sl96_ll48_pl96_dm512_el3_dl1_df512_fc1_ebtimeF_dtTrue_Exp>>>>>>>>>>>>>>>>>>>>>>>>>> train 36696 val 5175 test 10444 iters: 100, epoch: 1 | loss: 0.4036454 speed: 0.0235s/iter; left time: 537.3747s iters: 200, epoch: 1 | loss: 0.3538151 speed: 0.0142s/iter; left time: 323.6224s iters: 300, epoch: 1 | loss: 0.3341010 speed: 0.0138s/iter; left time: 312.5898s iters: 400, epoch: 1 | loss: 0.3313734 speed: 0.0138s/iter; left time: 312.0042s iters: 500, epoch: 1 | loss: 0.3725422 speed: 0.0149s/iter; left time: 333.6391s iters: 600, epoch: 1 | loss: 2.2886326 speed: 0.0152s/iter; left time: 339.5225s iters: 700, epoch: 1 | loss: 0.4027846 speed: 0.0166s/iter; left time: 368.6571s iters: 800, epoch: 1 | loss: 0.3414143 speed: 0.0175s/iter; left time: 386.4240s iters: 900, epoch: 1 | loss: 0.4444550 speed: 0.0171s/iter; left time: 376.6025s iters: 1000, epoch: 1 | loss: 0.3779259 speed: 0.0175s/iter; left time: 383.3128s iters: 1100, epoch: 1 | loss: 0.3262408 speed: 0.0174s/iter; left time: 379.6096s iters: 1200, epoch: 1 | loss: 0.2622951 speed: 0.0175s/iter; left time: 379.7748s iters: 1300, epoch: 1 | loss: 0.5229304 speed: 0.0178s/iter; left time: 384.5851s iters: 1400, epoch: 1 | loss: 0.5858208 speed: 0.0192s/iter; left time: 414.5758s iters: 1500, epoch: 1 | loss: 0.5290294 speed: 0.0204s/iter; left time: 437.1839s iters: 1600, epoch: 1 | loss: 0.3211349 speed: 0.0204s/iter; left time: 434.5036s iters: 1700, epoch: 1 | loss: 0.3488162 speed: 0.0203s/iter; left time: 431.8828s iters: 1800, epoch: 1 | loss: 0.3670591 speed: 0.0203s/iter; left time: 430.0920s iters: 1900, epoch: 1 | loss: 0.2346632 speed: 0.0204s/iter; left time: 428.7342s iters: 2000, epoch: 1 | loss: 0.6296965 speed: 0.0204s/iter; left time: 426.3540s iters: 2100, epoch: 1 | loss: 0.3551054 speed: 0.0203s/iter; left time: 423.3616s iters: 2200, epoch: 1 | loss: 0.2917550 speed: 0.0203s/iter; left time: 421.3748s Epoch: 1 cost time: 41.27806258201599 Epoch: 1, Steps: 2294 | Train Loss: 0.4738744 Vali Loss: 0.4414336 Test Loss: 0.1900593 Validation loss decreased (inf --> 0.441434). Saving model ... Updating learning rate to 0.00029265847744427303 iters: 100, epoch: 2 | loss: 0.6401179 speed: 0.0706s/iter; left time: 1449.7168s iters: 200, epoch: 2 | loss: 0.3250378 speed: 0.0135s/iter; left time: 275.2172s iters: 300, epoch: 2 | loss: 0.2748268 speed: 0.0135s/iter; left time: 274.1024s iters: 400, epoch: 2 | loss: 0.6726312 speed: 0.0135s/iter; left time: 272.6539s iters: 500, epoch: 2 | loss: 0.2553286 speed: 0.0136s/iter; left time: 273.6538s iters: 600, epoch: 2 | loss: 0.4415316 speed: 0.0136s/iter; left time: 272.8549s iters: 700, epoch: 2 | loss: 0.4522270 speed: 0.0137s/iter; left time: 272.3512s iters: 800, epoch: 2 | loss: 0.4513119 speed: 0.0131s/iter; left time: 259.6996s iters: 900, epoch: 2 | loss: 0.4280919 speed: 0.0138s/iter; left time: 271.9492s iters: 1000, epoch: 2 | loss: 0.2037832 speed: 0.0138s/iter; left time: 270.6034s iters: 1100, epoch: 2 | loss: 0.3414104 speed: 0.0138s/iter; left time: 269.3662s iters: 1200, epoch: 2 | loss: 0.3679063 speed: 0.0138s/iter; left time: 268.3948s iters: 1300, epoch: 2 | loss: 2.2532523 speed: 0.0138s/iter; left time: 266.9653s iters: 1400, epoch: 2 | loss: 0.3440191 speed: 0.0160s/iter; left time: 307.6901s iters: 1500, epoch: 2 | loss: 0.3955808 speed: 0.0154s/iter; left time: 295.5054s iters: 1600, epoch: 2 | loss: 0.4907824 speed: 0.0138s/iter; left time: 263.0443s iters: 1700, epoch: 2 | loss: 2.2307138 speed: 0.0140s/iter; left time: 264.6075s iters: 1800, epoch: 2 | loss: 0.3165238 speed: 0.0161s/iter; left time: 303.6546s iters: 1900, epoch: 2 | loss: 0.3220946 speed: 0.0161s/iter; left time: 302.2597s iters: 2000, epoch: 2 | loss: 0.2449237 speed: 0.0170s/iter; left time: 317.1356s iters: 2100, epoch: 2 | loss: 0.3095628 speed: 0.0170s/iter; left time: 315.8705s iters: 2200, epoch: 2 | loss: 0.2945173 speed: 0.0170s/iter; left time: 313.6160s Epoch: 2 cost time: 33.299219608306885 Epoch: 2, Steps: 2294 | Train Loss: 0.5480080 Vali Loss: 0.4419284 Test Loss: 0.1754540 EarlyStopping counter: 1 out of 10 Updating learning rate to 0.0002713525491562421 iters: 100, epoch: 3 | loss: 0.2709894 speed: 0.0637s/iter; left time: 1163.6169s iters: 200, epoch: 3 | loss: 0.2794941 speed: 0.0165s/iter; left time: 300.3870s iters: 300, epoch: 3 | loss: 0.3516423 speed: 0.0165s/iter; left time: 298.4946s iters: 400, epoch: 3 | loss: 0.3174662 speed: 0.0155s/iter; left time: 278.3513s iters: 500, epoch: 3 | loss: 0.3157647 speed: 0.0163s/iter; left time: 290.3424s iters: 600, epoch: 3 | loss: 0.2292653 speed: 0.0165s/iter; left time: 292.6444s iters: 700, epoch: 3 | loss: 0.2290015 speed: 0.0165s/iter; left time: 291.4030s iters: 800, epoch: 3 | loss: 0.2470866 speed: 0.0165s/iter; left time: 290.1491s iters: 900, epoch: 3 | loss: 0.2827386 speed: 0.0165s/iter; left time: 288.1084s iters: 1000, epoch: 3 | loss: 0.3288465 speed: 0.0165s/iter; left time: 286.5521s iters: 1100, epoch: 3 | loss: 0.4127340 speed: 0.0174s/iter; left time: 300.6309s iters: 1200, epoch: 3 | loss: 0.2658502 speed: 0.0179s/iter; left time: 307.8953s iters: 1300, epoch: 3 | loss: 0.2911725 speed: 0.0179s/iter; left time: 305.8910s iters: 1400, epoch: 3 | loss: 0.2256956 speed: 0.0180s/iter; left time: 304.4094s iters: 1500, epoch: 3 | loss: 0.3485063 speed: 0.0179s/iter; left time: 301.6632s iters: 1600, epoch: 3 | loss: 0.3210509 speed: 0.0149s/iter; left time: 249.0080s iters: 1700, epoch: 3 | loss: 0.3115217 speed: 0.0136s/iter; left time: 226.7412s iters: 1800, epoch: 3 | loss: 0.4391301 speed: 0.0134s/iter; left time: 221.6049s iters: 1900, epoch: 3 | loss: 0.3646879 speed: 0.0136s/iter; left time: 222.9945s iters: 2000, epoch: 3 | loss: 0.2265420 speed: 0.0136s/iter; left time: 221.8785s iters: 2100, epoch: 3 | loss: 0.2624652 speed: 0.0164s/iter; left time: 267.0513s iters: 2200, epoch: 3 | loss: 0.8389145 speed: 0.0165s/iter; left time: 266.9960s Epoch: 3 cost time: 36.725884199142456 Epoch: 3, Steps: 2294 | Train Loss: 0.3254803 Vali Loss: 0.4416376 Test Loss: 0.1746636 EarlyStopping counter: 2 out of 10 Updating learning rate to 0.00023816778784387094 iters: 100, epoch: 4 | loss: 0.2669576 speed: 0.0619s/iter; left time: 987.0867s iters: 200, epoch: 4 | loss: 0.3026834 speed: 0.0144s/iter; left time: 228.4021s iters: 300, epoch: 4 | loss: 0.2467037 speed: 0.0166s/iter; left time: 262.0724s iters: 400, epoch: 4 | loss: 0.2076253 speed: 0.0166s/iter; left time: 260.3927s iters: 500, epoch: 4 | loss: 0.2368142 speed: 0.0166s/iter; left time: 258.9158s iters: 600, epoch: 4 | loss: 0.1926093 speed: 0.0167s/iter; left time: 257.4968s iters: 700, epoch: 4 | loss: 0.2499007 speed: 0.0166s/iter; left time: 254.2027s iters: 800, epoch: 4 | loss: 3.8899975 speed: 0.0175s/iter; left time: 267.6255s iters: 900, epoch: 4 | loss: 0.4037665 speed: 0.0185s/iter; left time: 280.3921s iters: 1000, epoch: 4 | loss: 0.2400860 speed: 0.0192s/iter; left time: 289.2192s iters: 1100, epoch: 4 | loss: 0.2650346 speed: 0.0193s/iter; left time: 288.0967s iters: 1200, epoch: 4 | loss: 0.2793321 speed: 0.0200s/iter; left time: 297.8566s iters: 1300, epoch: 4 | loss: 0.2470463 speed: 0.0161s/iter; left time: 237.5367s iters: 1400, epoch: 4 | loss: 0.2521619 speed: 0.0139s/iter; left time: 203.9402s iters: 1500, epoch: 4 | loss: 0.2568092 speed: 0.0148s/iter; left time: 215.6603s iters: 1600, epoch: 4 | loss: 0.2051758 speed: 0.0167s/iter; left time: 242.0249s iters: 1700, epoch: 4 | loss: 0.2405885 speed: 0.0176s/iter; left time: 252.6166s iters: 1800, epoch: 4 | loss: 0.2696667 speed: 0.0188s/iter; left time: 267.8078s iters: 1900, epoch: 4 | loss: 0.1836121 speed: 0.0195s/iter; left time: 276.2933s iters: 2000, epoch: 4 | loss: 0.1976945 speed: 0.0205s/iter; left time: 287.6593s iters: 2100, epoch: 4 | loss: 0.3522930 speed: 0.0204s/iter; left time: 285.0292s iters: 2200, epoch: 4 | loss: 0.2470250 speed: 0.0203s/iter; left time: 281.6042s Epoch: 4 cost time: 40.027498960494995 Epoch: 4, Steps: 2294 | Train Loss: 0.4197084 Vali Loss: 0.4251894 Test Loss: 0.1686089 Validation loss decreased (0.441434 --> 0.425189). Saving model ... Updating learning rate to 0.0001963525491562421 iters: 100, epoch: 5 | loss: 0.3512476 speed: 0.0692s/iter; left time: 945.5842s iters: 200, epoch: 5 | loss: 0.2025531 speed: 0.0140s/iter; left time: 189.6152s iters: 300, epoch: 5 | loss: 0.3151135 speed: 0.0138s/iter; left time: 186.3076s iters: 400, epoch: 5 | loss: 0.1649124 speed: 0.0137s/iter; left time: 182.9132s iters: 500, epoch: 5 | loss: 0.1982203 speed: 0.0134s/iter; left time: 178.0392s iters: 600, epoch: 5 | loss: 0.3465936 speed: 0.0150s/iter; left time: 197.7103s iters: 700, epoch: 5 | loss: 0.2815661 speed: 0.0151s/iter; left time: 197.4270s iters: 800, epoch: 5 | loss: 0.3126697 speed: 0.0137s/iter; left time: 177.7352s iters: 900, epoch: 5 | loss: 0.2094218 speed: 0.0137s/iter; left time: 176.3205s iters: 1000, epoch: 5 | loss: 2.2910328 speed: 0.0138s/iter; left time: 175.7294s iters: 1100, epoch: 5 | loss: 0.1714488 speed: 0.0138s/iter; left time: 174.9900s iters: 1200, epoch: 5 | loss: 0.5156409 speed: 0.0138s/iter; left time: 173.4002s iters: 1300, epoch: 5 | loss: 0.1571699 speed: 0.0138s/iter; left time: 171.9752s iters: 1400, epoch: 5 | loss: 0.3079891 speed: 0.0138s/iter; left time: 170.7172s iters: 1500, epoch: 5 | loss: 0.2646823 speed: 0.0137s/iter; left time: 168.3070s iters: 1600, epoch: 5 | loss: 0.3137054 speed: 0.0137s/iter; left time: 166.5632s iters: 1700, epoch: 5 | loss: 0.2070766 speed: 0.0138s/iter; left time: 166.1373s iters: 1800, epoch: 5 | loss: 0.2626841 speed: 0.0162s/iter; left time: 193.9877s iters: 1900, epoch: 5 | loss: 0.2151637 speed: 0.0169s/iter; left time: 200.2149s iters: 2000, epoch: 5 | loss: 0.2760842 speed: 0.0173s/iter; left time: 203.8911s iters: 2100, epoch: 5 | loss: 0.2975026 speed: 0.0190s/iter; left time: 221.6073s iters: 2200, epoch: 5 | loss: 0.2145984 speed: 0.0190s/iter; left time: 219.5113s Epoch: 5 cost time: 34.31195425987244 Epoch: 5, Steps: 2294 | Train Loss: 0.3580489 Vali Loss: 0.4476494 Test Loss: 0.1756839 EarlyStopping counter: 1 out of 10 Updating learning rate to 0.00015 iters: 100, epoch: 6 | loss: 0.2923902 speed: 0.0681s/iter; left time: 773.8002s iters: 200, epoch: 6 | loss: 0.1475460 speed: 0.0136s/iter; left time: 153.8053s iters: 300, epoch: 6 | loss: 0.2080690 speed: 0.0135s/iter; left time: 151.2166s iters: 400, epoch: 6 | loss: 0.3924242 speed: 0.0136s/iter; left time: 151.0847s iters: 500, epoch: 6 | loss: 0.3908802 speed: 0.0137s/iter; left time: 150.8331s iters: 600, epoch: 6 | loss: 0.1254497 speed: 0.0098s/iter; left time: 106.4976s iters: 700, epoch: 6 | loss: 0.2812734 speed: 0.0097s/iter; left time: 104.0888s iters: 800, epoch: 6 | loss: 0.3186392 speed: 0.0097s/iter; left time: 103.0860s iters: 900, epoch: 6 | loss: 0.1809346 speed: 0.0094s/iter; left time: 99.6321s iters: 1000, epoch: 6 | loss: 0.1881777 speed: 0.0138s/iter; left time: 144.2360s iters: 1100, epoch: 6 | loss: 0.1834134 speed: 0.0137s/iter; left time: 141.6584s iters: 1200, epoch: 6 | loss: 0.2215981 speed: 0.0130s/iter; left time: 133.5223s iters: 1300, epoch: 6 | loss: 0.1686825 speed: 0.0144s/iter; left time: 146.0337s iters: 1400, epoch: 6 | loss: 0.2317298 speed: 0.0166s/iter; left time: 166.8591s iters: 1500, epoch: 6 | loss: 0.2422193 speed: 0.0165s/iter; left time: 164.9172s iters: 1600, epoch: 6 | loss: 2.1373572 speed: 0.0167s/iter; left time: 164.6783s iters: 1700, epoch: 6 | loss: 0.1828119 speed: 0.0174s/iter; left time: 169.8618s iters: 1800, epoch: 6 | loss: 0.2676963 speed: 0.0197s/iter; left time: 190.4514s iters: 1900, epoch: 6 | loss: 0.4651260 speed: 0.0197s/iter; left time: 188.9532s iters: 2000, epoch: 6 | loss: 0.1579365 speed: 0.0157s/iter; left time: 148.9484s iters: 2100, epoch: 6 | loss: 0.1624753 speed: 0.0136s/iter; left time: 127.4397s iters: 2200, epoch: 6 | loss: 0.2036208 speed: 0.0160s/iter; left time: 148.4296s Epoch: 6 cost time: 32.924118757247925 Epoch: 6, Steps: 2294 | Train Loss: 0.3250205 Vali Loss: 0.4720847 Test Loss: 0.1833609 EarlyStopping counter: 2 out of 10 Updating learning rate to 0.0001036474508437579 iters: 100, epoch: 7 | loss: 0.2263691 speed: 0.0598s/iter; left time: 542.5842s iters: 200, epoch: 7 | loss: 0.1706614 speed: 0.0147s/iter; left time: 132.3575s iters: 300, epoch: 7 | loss: 0.4450584 speed: 0.0150s/iter; left time: 133.5679s iters: 400, epoch: 7 | loss: 0.1596253 speed: 0.0149s/iter; left time: 130.5566s iters: 500, epoch: 7 | loss: 0.1552347 speed: 0.0149s/iter; left time: 129.5256s iters: 600, epoch: 7 | loss: 0.1781175 speed: 0.0143s/iter; left time: 122.5565s iters: 700, epoch: 7 | loss: 0.2057821 speed: 0.0150s/iter; left time: 127.2036s iters: 800, epoch: 7 | loss: 0.1817286 speed: 0.0154s/iter; left time: 128.6435s iters: 900, epoch: 7 | loss: 0.1354985 speed: 0.0162s/iter; left time: 134.4697s iters: 1000, epoch: 7 | loss: 0.3534313 speed: 0.0163s/iter; left time: 132.8846s iters: 1100, epoch: 7 | loss: 2.1183684 speed: 0.0163s/iter; left time: 131.8062s iters: 1200, epoch: 7 | loss: 2.1941707 speed: 0.0184s/iter; left time: 146.6815s iters: 1300, epoch: 7 | loss: 0.1697403 speed: 0.0196s/iter; left time: 154.2662s iters: 1400, epoch: 7 | loss: 0.2164193 speed: 0.0196s/iter; left time: 152.4956s iters: 1500, epoch: 7 | loss: 0.1487056 speed: 0.0192s/iter; left time: 147.7702s iters: 1600, epoch: 7 | loss: 0.4190980 speed: 0.0188s/iter; left time: 142.2020s iters: 1700, epoch: 7 | loss: 0.2058418 speed: 0.0188s/iter; left time: 140.8787s iters: 1800, epoch: 7 | loss: 0.1226447 speed: 0.0194s/iter; left time: 142.8435s iters: 1900, epoch: 7 | loss: 0.2356446 speed: 0.0205s/iter; left time: 148.8279s iters: 2000, epoch: 7 | loss: 0.2367059 speed: 0.0206s/iter; left time: 147.8381s iters: 2100, epoch: 7 | loss: 0.1676833 speed: 0.0206s/iter; left time: 145.6715s iters: 2200, epoch: 7 | loss: 0.1725331 speed: 0.0199s/iter; left time: 139.0891s Epoch: 7 cost time: 39.72669458389282 Epoch: 7, Steps: 2294 | Train Loss: 0.3917756 Vali Loss: 0.4726515 Test Loss: 0.1887089 EarlyStopping counter: 3 out of 10 Updating learning rate to 6.183221215612904e-05 iters: 100, epoch: 8 | loss: 0.1433359 speed: 0.0677s/iter; left time: 459.1125s iters: 200, epoch: 8 | loss: 0.2055279 speed: 0.0139s/iter; left time: 93.1049s iters: 300, epoch: 8 | loss: 0.2131681 speed: 0.0163s/iter; left time: 107.2426s iters: 400, epoch: 8 | loss: 0.2196651 speed: 0.0163s/iter; left time: 105.5213s iters: 500, epoch: 8 | loss: 0.3819869 speed: 0.0163s/iter; left time: 103.9812s iters: 600, epoch: 8 | loss: 0.1870438 speed: 0.0163s/iter; left time: 102.2659s iters: 700, epoch: 8 | loss: 0.0745629 speed: 0.0138s/iter; left time: 85.6146s iters: 800, epoch: 8 | loss: 0.2042980 speed: 0.0138s/iter; left time: 84.0793s iters: 900, epoch: 8 | loss: 0.1475536 speed: 0.0139s/iter; left time: 83.4538s iters: 1000, epoch: 8 | loss: 0.4048573 speed: 0.0140s/iter; left time: 82.2775s iters: 1100, epoch: 8 | loss: 0.2045363 speed: 0.0140s/iter; left time: 80.7272s iters: 1200, epoch: 8 | loss: 0.2384610 speed: 0.0140s/iter; left time: 79.3232s iters: 1300, epoch: 8 | loss: 0.1296934 speed: 0.0161s/iter; left time: 89.8580s iters: 1400, epoch: 8 | loss: 0.2461748 speed: 0.0177s/iter; left time: 96.8697s iters: 1500, epoch: 8 | loss: 0.1433063 speed: 0.0177s/iter; left time: 95.1735s iters: 1600, epoch: 8 | loss: 2.0703864 speed: 0.0177s/iter; left time: 93.7113s iters: 1700, epoch: 8 | loss: 0.3135197 speed: 0.0195s/iter; left time: 100.8796s iters: 1800, epoch: 8 | loss: 0.2275418 speed: 0.0197s/iter; left time: 100.1432s iters: 1900, epoch: 8 | loss: 0.2258315 speed: 0.0207s/iter; left time: 102.9563s iters: 2000, epoch: 8 | loss: 0.5402485 speed: 0.0206s/iter; left time: 100.6643s iters: 2100, epoch: 8 | loss: 0.1666155 speed: 0.0206s/iter; left time: 98.3946s iters: 2200, epoch: 8 | loss: 0.3003441 speed: 0.0204s/iter; left time: 95.3980s Epoch: 8 cost time: 38.66363549232483 Epoch: 8, Steps: 2294 | Train Loss: 0.3176663 Vali Loss: 0.4808291 Test Loss: 0.1920194 EarlyStopping counter: 4 out of 10 Updating learning rate to 2.8647450843757897e-05 iters: 100, epoch: 9 | loss: 0.5826761 speed: 0.0683s/iter; left time: 306.4070s iters: 200, epoch: 9 | loss: 0.1596118 speed: 0.0138s/iter; left time: 60.7506s iters: 300, epoch: 9 | loss: 0.2060688 speed: 0.0138s/iter; left time: 59.1969s iters: 400, epoch: 9 | loss: 0.2674173 speed: 0.0138s/iter; left time: 57.8440s iters: 500, epoch: 9 | loss: 0.1885380 speed: 0.0129s/iter; left time: 52.7814s iters: 600, epoch: 9 | loss: 0.1056275 speed: 0.0125s/iter; left time: 50.0567s iters: 700, epoch: 9 | loss: 0.0917342 speed: 0.0126s/iter; left time: 49.1035s iters: 800, epoch: 9 | loss: 2.0841510 speed: 0.0126s/iter; left time: 47.7344s iters: 900, epoch: 9 | loss: 0.3349448 speed: 0.0126s/iter; left time: 46.4777s iters: 1000, epoch: 9 | loss: 0.1489390 speed: 0.0128s/iter; left time: 46.1043s iters: 1100, epoch: 9 | loss: 0.1504216 speed: 0.0128s/iter; left time: 44.7529s iters: 1200, epoch: 9 | loss: 0.1942857 speed: 0.0127s/iter; left time: 43.0960s iters: 1300, epoch: 9 | loss: 0.0933912 speed: 0.0127s/iter; left time: 41.7903s iters: 1400, epoch: 9 | loss: 0.1358986 speed: 0.0127s/iter; left time: 40.3545s iters: 1500, epoch: 9 | loss: 0.3633728 speed: 0.0120s/iter; left time: 37.0319s iters: 1600, epoch: 9 | loss: 0.1545264 speed: 0.0130s/iter; left time: 38.9192s iters: 1700, epoch: 9 | loss: 0.1448196 speed: 0.0135s/iter; left time: 39.0197s iters: 1800, epoch: 9 | loss: 0.2245946 speed: 0.0137s/iter; left time: 38.2689s iters: 1900, epoch: 9 | loss: 0.1365054 speed: 0.0137s/iter; left time: 36.7369s iters: 2000, epoch: 9 | loss: 0.4193476 speed: 0.0137s/iter; left time: 35.5790s iters: 2100, epoch: 9 | loss: 0.2448702 speed: 0.0138s/iter; left time: 34.2597s iters: 2200, epoch: 9 | loss: 0.1035335 speed: 0.0138s/iter; left time: 33.0561s Epoch: 9 cost time: 30.47970199584961 Epoch: 9, Steps: 2294 | Train Loss: 0.2970580 Vali Loss: 0.5224582 Test Loss: 0.1955109 EarlyStopping counter: 5 out of 10 Updating learning rate to 7.34152255572697e-06 iters: 100, epoch: 10 | loss: 0.0942249 speed: 0.0648s/iter; left time: 142.1427s iters: 200, epoch: 10 | loss: 0.1276101 speed: 0.0140s/iter; left time: 29.3143s iters: 300, epoch: 10 | loss: 0.1668937 speed: 0.0154s/iter; left time: 30.7609s iters: 400, epoch: 10 | loss: 0.2278446 speed: 0.0172s/iter; left time: 32.6500s iters: 500, epoch: 10 | loss: 0.4736909 speed: 0.0175s/iter; left time: 31.4656s iters: 600, epoch: 10 | loss: 0.1455495 speed: 0.0143s/iter; left time: 24.2843s iters: 700, epoch: 10 | loss: 0.5455921 speed: 0.0141s/iter; left time: 22.5467s iters: 800, epoch: 10 | loss: 0.0883168 speed: 0.0141s/iter; left time: 21.0407s iters: 900, epoch: 10 | loss: 0.2184256 speed: 0.0145s/iter; left time: 20.2428s iters: 1000, epoch: 10 | loss: 0.1521411 speed: 0.0149s/iter; left time: 19.3338s iters: 1100, epoch: 10 | loss: 0.2715932 speed: 0.0138s/iter; left time: 16.4973s iters: 1200, epoch: 10 | loss: 0.2269915 speed: 0.0138s/iter; left time: 15.1206s iters: 1300, epoch: 10 | loss: 0.1844854 speed: 0.0138s/iter; left time: 13.7492s iters: 1400, epoch: 10 | loss: 0.4068148 speed: 0.0138s/iter; left time: 12.3498s iters: 1500, epoch: 10 | loss: 0.1411258 speed: 0.0138s/iter; left time: 10.9729s iters: 1600, epoch: 10 | loss: 1.9868090 speed: 0.0138s/iter; left time: 9.5946s iters: 1700, epoch: 10 | loss: 0.1957721 speed: 0.0138s/iter; left time: 8.1814s iters: 1800, epoch: 10 | loss: 0.1239839 speed: 0.0137s/iter; left time: 6.8046s iters: 1900, epoch: 10 | loss: 0.1818440 speed: 0.0138s/iter; left time: 5.4356s iters: 2000, epoch: 10 | loss: 0.2910723 speed: 0.0138s/iter; left time: 4.0713s iters: 2100, epoch: 10 | loss: 0.1157951 speed: 0.0138s/iter; left time: 2.6874s iters: 2200, epoch: 10 | loss: 0.2038301 speed: 0.0138s/iter; left time: 1.3093s Epoch: 10 cost time: 32.87806439399719 Epoch: 10, Steps: 2294 | Train Loss: 0.2986548 Vali Loss: 0.5184059 Test Loss: 0.1979387 EarlyStopping counter: 6 out of 10 Updating learning rate to 0.0 testing : long_term_forecast_weather_96_96_SOFTS_custom_ftM_sl96_ll48_pl96_dm512_el3_dl1_df512_fc1_ebtimeF_dtTrue_Exp<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< test 10444 mse:0.1686088588118713, mae:0.20914341695678046

[hanlu-nju]

Yes, some datasets are hard to predict due to their inherent non-stationarity. Our method may exhibit overfitting and achieve lower test loss in the early epochs. It is a common phenomenon for some time series data. Our approach aims to provide a better general solution on average. For special cases, like weather, we recommend making targeted adjustments, e.g., weight decay, robust loss, or certain preprocessing.