Paper reproduction

[Shan2L]

Hi authors, Thanks for your efforts! However, I have encountered some problems when I tried to reproduce the experiment in paper with ETTh1 dataset. To be specific, the MAE and MSE reported in paper is 0.424 and 0.400 respectively. But the result of my reproduction is as followed: MSE[mean]: 11.36348028399042 MSE[0.5]: 11.251827251274198 MAE[0.5]: 1.9022683191148606 MASE[0.5]: 1.2261620553695651 MAPE[0.5]: 0.8144727617279135 sMAPE[0.5]: 0.495835838503763 MSIS: 9.147484712758356 RMSE[mean]: 3.370976161884035 NRMSE[mean]: 0.6734112487403957 ND[0.5]: 0.3800112557006269 mean_weighted_sum_quantile_loss: 0.30314495703261496

and I just follow the step give in readme:

1. python -m uni2ts.data.builder.simple ETTh1 dataset/ETT-small/ETTh1.csv --date_offset '2017-10-23 23:00:00'
2. python -m cli.eval run_name=example_eval_1 model=moirai_1.0_R_small model.patch_size=32 model.context_length=1000 data=etth1_test

I just wonder what the problem might be? Looking forward to your reply. Thanks so much!

[gorold]

Hi, the results in the paper for ETTh1 follows the long sequence time series forecasting benchmark setting (check out the Informer and Autoformer papers), which normalize the whole time series based on the train set's mean and std. You can use the lsf_test config to replicate the results.

[zqiao11]

Hi @gorold. For paper reproduction, I checked the branch feat/reproduce-project and ran some experiments on ETTh1 with your provided configurations. For Moirai-small with prediction length of 96, the MAE is the same as your original result (Table 22). But for MSE, it is slightly different. Mine is 0.387 and yours is 0.375.

Is it normal to have such difference when using the same config for evaluation?

[gorold]

data.mode=M is required too

[zqiao11]

Thank you very much. Now the results are exactly the same.

[Shan2L]

@gorold Thanks for your reply. I have checked another branch 'feat/reproduce-project'. After I ran the pf_smalll.sh script, the code works normally for electricity and solar, but when came to others, an error occured, which said: (uni2ts) ➜ uni2ts git:(feat/reproduce-project) ✗ HYDRA_FULL_ERROR=1 python -m cli.eval \ run_name=pf_eval \ model=moirai_1.0_R_small \ model.patch_size=32 \ model.context_length=1000 \ data=gluonts_test \ data.dataset_name=walmart \ data.regenerate=True [2024-04-19 15:11:18,899][datasets][INFO] - PyTorch version 2.2.2 available. [2024-04-19 15:11:18,899][datasets][INFO] - JAX version 0.4.26 available. [DEBUG] The dataset name iswalmart [2024-04-19 15:11:19,035][root][INFO] - downloading and processing walmart Error executing job with overrides: ['run_name=pf_eval', 'model=moirai_1.0_R_small', 'model.patch_size=32', 'model.context_length=1000', 'data=gluonts_test', 'data.dataset_name=walmart', 'data.regenerate=True'] Traceback (most recent call last): File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 92, in _call_target return target(*args, kwargs) File "/home/shanlin/exp/uni2ts/src/uni2ts/eval_util/data.py", line 71, in get_gluonts_test_dataset dataset = get_dataset( File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/gluonts/dataset/repository/datasets.py", line 360, in get_dataset dataset_path = materialize_dataset( File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/gluonts/dataset/repository/datasets.py", line 307, in materialize_dataset dataset_recipe(kwargs) File "/home/shanlin/exp/uni2ts/src/uni2ts/eval_util/_pf_dataset.py", line 141, in generate_pf_dataset data, start, freq, prediction_length, rolling_evaluations = load_func( File "/home/shanlin/exp/uni2ts/src/uni2ts/eval_util/_pf_dataset.py", line 52, in _load_walmart os.path.join( File "/root/miniconda3/envs/uni2ts/lib/python3.10/posixpath.py", line 76, in join a = os.fspath(a) TypeError: expected str, bytes or os.PathLike object, not NoneType

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/root/miniconda3/envs/uni2ts/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/root/miniconda3/envs/uni2ts/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/home/shanlin/exp/uni2ts/cli/eval.py", line 73, in main() File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/main.py", line 90, in decorated_main _run_hydra( File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra _run_app( File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app run_and_report( File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/_internal/utils.py", line 222, in run_and_report raise ex File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/_internal/utils.py", line 219, in run_and_report return func() File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in lambda: hydra.run( File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/internal/hydra.py", line 132, in run = ret.return_value File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value raise self._return_value File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job ret.return_value = task_function(task_cfg) File "/home/shanlin/exp/uni2ts/cli/eval.py", line 30, in main test_data, metadata = call(cfg.data) File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 226, in instantiate return instantiate_node( File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 347, in instantiate_node return _call_target(target, partial, args, kwargs, full_key) File "/root/miniconda3/envs/uni2ts/lib/python3.10/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 97, in _call_target raise InstantiationException(msg) from e hydra.errors.InstantiationException: Error in call to target 'uni2ts.eval_util.data.get_gluonts_test_dataset': TypeError('expected str, bytes or os.PathLike object, not NoneType') full_key: data

After I dived into the source code, I found that might attribute to the absence of LSF_PATH/walmart-recruiting-store-sales-forecasting/train.csv, from which our dataset will be made.

I just want to make sure is that the root cause of the error? And can you tell me where should I find the missing datasets if possible?

Thanks so much!

[gorold]

Some datasets are not provided by the gluonts library, and need to be added the LSF_Path.

• Walmart: https://www.kaggle.com/competitions/walmart-recruiting-store-sales-forecasting/overview
• Istanbul Traffic: https://www.kaggle.com/datasets/leonardo00/istanbul-traffic-index
• Turkey Power: https://www.kaggle.com/datasets/dharanikra/electrical-power-demand-in-turkey

[Shan2L]

I see. Thanks for your kindly reply. This issue will be cloesd.

[gorold]

Re-opening to track the update for the additional datasets

[shiyuxin2000]

Hi @gorold, Thanks for efforts. Now I want to reproduce other experiments like results on Monash dataset, and probabilistic forecasting. Could you please offer some scripts on other datasets (like Monash)? Also, for LTSF experiments, how should I set the context length? Thank you so much

[Shan2L]

@gorold hi, I just want to make sure that mode=S stands for univariate and mode=M stands for multi-variate, right? Thanks!

[gorold]

@shiyuxin2000 You can refer to this page https://github.com/SalesforceAIResearch/uni2ts/tree/feat/reproduce-project/project/moirai-1 for now, we'll merge it to the main branch soon.

@Shan2L Yup, thats right.

[zqiao11]

Hi @gorold. May I further clarify regarding the Moirai's mode? Is mode=S using channel independence assumption for forecasting? And mode=M means flattening MTS into 1d sequence and using Any-variate attention?

Thanks.

[gorold]

@zqiao11 sorry for the late response, mode=S converts a multivariate time series with N variates into N univariate time series.