Reproducing Results

[camerons1967]

Hello,

I downloaded the repository to my computer and tried to reproduce the results that were published in the paper for the traffic dataset with a prediction window of length 96. I ran the code with the following args:

--hyperopt_max_evals 10 --experiment_id run_1

But the results were 0.504 for the MSE and 0.311 for the MAE which is significantly worse than what I was expecting to achieve. Is there anything else that needs to be done before running the code and training the model in order to reproduce the results?

Thanks in advance!

[kdgutier]

Hi @camerons1967,

We would recommend you to try this in a Google Colab, it is a lot faster. Here is a link to the ETT's replication.

Eventually hyperopt bayesian optimization can still stuck in bad local optimas, so you need to increase the number of hyperparameter exploration steps.

[omerlux]

Hi, I've tried to work with the notebook and the code. The notebook reproduces the results, but the code doesn't. What is the reason for this issue? Is there something missing in the repository?

Thanks!

[kdgutier]

Hi @omerlux,

We conducted all of our experiments in GPUs (a lot faster). An explanation is that initialization of the network's parameters and rounding differ between GPUs and CPUs.

Nevertheless N-HiTS should be able to achieve similar results on the CPU if you let the hyperparameter optimization algorithm run for longer to escape bad local optimas.