search

intel-analytics / analytics-zoo

Distributed Tensorflow, Keras and PyTorch on Apache Spark/Flink & Ray
https://analytics-zoo.readthedocs.io/
Apache License 2.0
11 stars 3 forks source link

Chronos: When the dataset is multivariate, Auto Model and Autotsest may have negative improvements. #35

Open liangs6212 opened 2 years ago

liangs6212 commented 2 years ago

When I used forecaster\automodel\autotsest in the same file and tested it six times with the parameters unchanged, there were 3 negative improvements and 3 positive improvements. P.S. The dataset is network_traffic, and there is no similar situation on the univariate dataset. log path: cpx2-->/home/cpx/liangs/mul_single_log.txt

# TCNForecaster mse(AvgRate)  28026.504694819632
# TCNForecaster mse(total)  2.283820957717442e+22
# AutoTCN mse(AvgRate)  15804.34013031733 improve by  43.609307323868414 %
# AutoTCN mse(total)  1.2805747653238046e+22 improve by  43.92840817943666 %
# AutoTSEstimator mse(AvgRate)  11827.337451242214 improve by  57.799455979152626 %
# AutoTSEstimator mse(total)  9.407867757317523e+21 improve by  58.80645667285501 %

# TCNForecaster mse(AvgRate)  14584.228766034217
# TCNForecaster mse(total)  1.1828622532188679e+22
# AutoTCN mse(AvgRate)  15945.982666076641 improve by  -9.337167716498428 %
# AutoTCN mse(total)  1.30109186373673e+22 improve by  -9.995213744975754 %
# AutoTSEstimator mse(AvgRate)  18524.652257574562 improve by  -27.018387840413965 %
# AutoTSEstimator mse(total)  1.5047155456221437e+22 improve by  -27.20970185052668 %
TheaperDeng commented 2 years ago

The issue may caused by your param setting in AutoTSEstimator

future_seq_len=hp.randint(5, 10*2),

This may make the forecasting task to change (and it should not happen).

Still this give us an alarm that we need to give a runtime error when user give an hp sampling function to future_seq_len

related doc: https://analytics-zoo.readthedocs.io/en/latest/doc/PythonAPI/Chronos/autotsestimator.html#autotsestimator