Hi, this is an impressive project, and thanks for sharing w/ community!
I've been trying to learn the model with my test data. The test data has about 70 samples, each with about 2300 timesteps.
However, in final stage, the Recovery Loss in Rolling Validation is getting bigger and bigger each round, and early stopped at 0.308, which caused the much worse wape and wape_global metrics than the baseline:
{'wape': 0.39331427, 'mape': 0.36864823, 'smape': 0.4937316, 'mae': 5.487852, 'rmse': 8.775283, 'nrmse': 0.47228432, 'wape_global': 0.582235, 'mape_global': 0.56812644, 'smape_global': 0.84549224, 'mae_global': 8.123833, 'rmse_global': 11.685119, 'nrmse_global': 0.47228432, 'baseline_wape': 0.11834013, 'baseline_mape': 0.11296055, 'baseline_smape': 0.11496856}
Could you provide some insights how can I improve the training and get better resulsts?
In addition, Factorization Loss F, Factorization Loss X, Validation Loss ended at (0.214, 0.205, 0.294) level, early stopped, while Temporal Loss hovered around 0.017 level.
Training of Xseq and Yseq, (training loss, validation loss) progress down to (0.074, 0.052) and (0.054, 0.021) respectively.
Hi, this is an impressive project, and thanks for sharing w/ community!
I've been trying to learn the model with my test data. The test data has about 70 samples, each with about 2300 timesteps.
However, in final stage, the Recovery Loss in Rolling Validation is getting bigger and bigger each round, and early stopped at 0.308, which caused the much worse wape and wape_global metrics than the baseline: {'wape': 0.39331427, 'mape': 0.36864823, 'smape': 0.4937316, 'mae': 5.487852, 'rmse': 8.775283, 'nrmse': 0.47228432, 'wape_global': 0.582235, 'mape_global': 0.56812644, 'smape_global': 0.84549224, 'mae_global': 8.123833, 'rmse_global': 11.685119, 'nrmse_global': 0.47228432, 'baseline_wape': 0.11834013, 'baseline_mape': 0.11296055, 'baseline_smape': 0.11496856}
Could you provide some insights how can I improve the training and get better resulsts?
Thanks!!!
===========================================================
Last round of Recovery Loss stats: GLO: rolling_validation(): Current window wape: 0.5014139 GLO: recover_future_X(): Recovery Loss(0/100000): 1.002367615699768 GLO: recover_future_X(): Recovery Loss(1000/100000): 0.628299355506897 GLO: recover_future_X(): Recovery Loss(2000/100000): 0.4282535910606384 GLO: recover_future_X(): Recovery Loss(3000/100000): 0.3461550176143646 GLO: recover_future_X(): Recovery Loss(4000/100000): 0.3201618790626526 GLO: recover_future_X(): Recovery Loss(5000/100000): 0.310817688703537 GLO: recover_future_X(): Recovery Loss(6000/100000): 0.3080223500728607 GLO: recover_future_X(): Recovery Loss(7000/100000): 0.3077664375305176 GLO: rolling_validation(): Current window wape: 0.45383096
In addition, Factorization Loss F, Factorization Loss X, Validation Loss ended at (0.214, 0.205, 0.294) level, early stopped, while Temporal Loss hovered around 0.017 level.
Training of Xseq and Yseq, (training loss, validation loss) progress down to (0.074, 0.052) and (0.054, 0.021) respectively.