Template Eval Error for most of the powerful and effective models

[sdkulkarniB]

Template Eval Error for most of the powerful and effective models including LightGBM, Xgboost, Prophet, GluonTS, etc.

The Template Eval Error just keeps on varying everytime. Wish there was one proper format for Formating the Dataset so that it could run effortless for all model inside AutoTS.

The aim was to push the limits to build a powerful forecasting model for specific time series.

Params included:

model = AutoTS( ensemble='all', model_list='all',
max_generations=15, num_validations=90, validation_method="even", )

Had to end execution in between because of Google Colab Runtime Error (Free Plan)

Logs is too big have been attached as file below... Logs.txt

[winedarksea]

Template eval errors are fairly normal. You can't expect every possible model parameter set to work on every possible dataset. Go into the model results and see what percentage of the models failed.

That said, it is rather suspicious that all of the models that you listed as failing are the ones which are not installed by default. Are you making sure that xgboost, lightgbm, prophet, and gluonts are all installed in the environment? In the log you posted, GluonTS is failing because it isn't installed correctly. Only pandas/numpy/scikit-learn/statsmodels are installed by default.

pip install autots["additional"] might help as that should install many of the optional dependencies

num_validations=90 is pretty high as well. Do you really intend to do 90 holdouts and evaluations? That's a lot. I think maybe you meant forecast_length=90? Also validation_method='even' is usually my last choice, 'backwards', 'seasonal', and 'similarity' are all usually more effective.

If you really want to push the limits, you are going to want to be doing more than 15 max_generations. You can set a generation_timeout also instead for number of minutes to stop searching. Generally you are going to find that doing a deeper search on a faster model list (scalable) will be more effective than a shallow search on a bigger list. Also I recommend stepping away from all and doing at most all_pragmatic as a the model_list as some of the models in all are of limited value.

Also make sure to try an export_template and an import_template so that you can speed up the search on future runs. In Colab you should be able to export the template to your google drive and load from there, so that it persists even if you lose your session.

Also you may find that it's not just how powerful the model is but which model you decide to choose that makes the most difference, and that tuning metric_weighting may be helpful for your use case.

There is one proper format of data for AutoTS, it's the wide style data, but long style is also supported for import as that is also common.

[sdkulkarniB]

Hey @winedarksea Thanks for your insights! Your answer clears a lot of queries.

(1) I tried installing autots['additional'] on Local Machine and yes it did give an installation error. Maybe it didn't got installed properly on Colab too. Using Colab Local Runtime with Anaconda + Jupyter Notebook. Environment Variables Path was not defined. Solved That!

(2) Yes I actually did num_validations = 90, I was focusing more on the accuracy so that was it. My dataset is just 3800 rows long. Day series. What should be approx value for Seasonal N? as in docs it said is more effective than all others. I was thinking n=60 something. The more validations the more accuracy, that what I thought. Correct me if I am wrong.

(3) As for the metric weighting should it be explicitly defined or Auto will do, coz I am not from stats background, I just used what was given in extended tutorial and it made sense after reading what was what. metric_weighting = { 'smape_weighting': 5, 'mae_weighting': 2, 'rmse_weighting': 2, 'made_weighting': 0.5, 'mage_weighting': 1, 'mle_weighting': 0, 'imle_weighting': 0, 'spl_weighting': 3, 'containment_weighting': 0, 'contour_weighting': 1, 'runtime_weighting': 0.05, }

4) I am using Wide Format - Date, Open, High, Low, Close, Volume, thats it.

Thanks again! And one question - Is AutoTS participating in M7 Competition? Would love to see its full potential there.:D

[winedarksea]

2: You want your validation samples to be as representative as possible of the future you are about to predict. More is better up to a point, but not always. For seasonal data, you often want to validate on the year before of the same time. So for daily data with yearly seasonality, you can do the auto "seasonal" or "seasonal 364". This would mean if you were forecasting on, say May, you would use May from the previous year (2023) and (2022) and so on for each new validation. This is most useful with cases like demand forecasting. A major limit on the number of validations is how much data history you have. For stock prices, the default 'backwards' will generally be best.

3. Your metric_weighting is tailored to the use case you are working on. Some cases you might care about long term trend, other times you might care about the shape of the forecasts, other times you might care about probabilistic forecasts. Metric helps you tell it what to focus on. If your forecasts look good with that default, then don't worry about it too much. If you tell me more about the needs of your use case, though, I might be able to suggest some different metrics. Since it seems like you are forecasting stock prices, then you might want to try 'dwae_weighting' and 'oda_weighting' which I used as part of the M6. These are about directional accuracy, predicting long vs short.

Yes, I'll definitely participate in the M7. I haven't seen any details about it yet. The word from organizers was that they were hoping to do energy (demand?) for that one.