search

winedarksea / AutoTS

Automated Time Series Forecasting
MIT License
1.1k stars 100 forks source link

Evaluation Metrics are missing and all models have failed, by an error in template or metrics #130

Closed tungsontran closed 1 year ago

tungsontran commented 2 years ago

Describe the bug I tried to use DNN models presented in the package such as GluonTS, pytorch-forecasting and tensorflowSTS. I have installed all required packages as in the installation guide (My environment can be viewed below). The error I got from the fit() function read as:

"Evaluation Metrics are missing and all models have failed, by an error in template or metrics. There are many possible causes for this, bad parameters, environment, or an unreported bug. Usually this means you are missing required packages for the models like fbprophet or gluonts, or that the models in model_list are inappropriate for your data. A new starting template may also help".

To Reproduce When I run the following code:

from autots import AutoTS
model_list = ['TensorflowSTS']
model = AutoTS(
    forecast_length=7,
    frequency='D',
    prediction_interval=0.9,
    metric_weighting=metric_weighting,
    ensemble='horizontal',
    holiday_country='UK',
    model_list=model_list,  
    drop_most_recent=0,
    max_generations=1,
    num_validations=3,
    validation_method="backwards",
    no_negatives=True,
    verbose=-2,
    n_jobs = 72
)

model = model.fit(train_df)

Expected behavior Stack trace:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/pandas/core/indexes/base.py:3621, in Index.get_loc(self, key, method, tolerance)
   3620 try:
-> 3621     return self._engine.get_loc(casted_key)
   3622 except KeyError as err:

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/pandas/_libs/index.pyx:136, in pandas._libs.index.IndexEngine.get_loc()

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/pandas/_libs/index.pyx:163, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:5198, in pandas._libs.hashtable.PyObjectHashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:5206, in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'smape'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/autots/evaluator/auto_model.py:2031, in generate_score(model_results, metric_weighting, prediction_interval)
   2030 # not sure why there are negative SMAPE values, but make sure they get dealt with
-> 2031 if model_results['smape'].min() < 0:
   2032     model_results['smape'] = model_results['smape'].where(
   2033         model_results['smape'] >= 0, model_results['smape'].max()
   2034     )

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/pandas/core/frame.py:3505, in DataFrame.__getitem__(self, key)
   3504     return self._getitem_multilevel(key)
-> 3505 indexer = self.columns.get_loc(key)
   3506 if is_integer(indexer):

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/pandas/core/indexes/base.py:3623, in Index.get_loc(self, key, method, tolerance)
   3622 except KeyError as err:
-> 3623     raise KeyError(key) from err
   3624 except TypeError:
   3625     # If we have a listlike key, _check_indexing_error will raise
   3626     #  InvalidIndexError. Otherwise we fall through and re-raise
   3627     #  the TypeError.

KeyError: 'smape'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
File <timed exec>:44, in <module>

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/autots/evaluator/auto_ts.py:791, in AutoTS.fit(self, df, date_col, value_col, id_col, future_regressor, weights, result_file, grouping_ids, validation_indexes)
    789 # capture the data from the lower level results
    790 self.initial_results = self.initial_results.concat(template_result)
--> 791 self.initial_results.model_results['Score'] = generate_score(
    792     self.initial_results.model_results,
    793     metric_weighting=metric_weighting,
    794     prediction_interval=prediction_interval,
    795 )
    796 if result_file is not None:
    797     self.initial_results.save(result_file)

File ~/anaconda3/envs/forecast/lib/python3.10/site-packages/autots/evaluator/auto_model.py:2126, in generate_score(model_results, metric_weighting, prediction_interval)
   2123         overall_score = overall_score + (containment_score * containment_weighting)
   2125 except Exception as e:
-> 2126     raise KeyError(
   2127         f"""Evaluation Metrics are missing and all models have failed, by an error in template or metrics.
   2128         There are many possible causes for this, bad parameters, environment, or an unreported bug.
   2129         Usually this means you are missing required packages for the models like fbprophet or gluonts,
   2130         or that the models in model_list are inappropriate for your data.
   2131         A new starting template may also help. {repr(e)}"""
   2132     )
   2134 return overall_score.astype(float)

Desktop (please complete the following information):

``

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge absl-py 1.1.0 pyhd8ed1ab_0 conda-forge aiohttp 3.8.1 py310h5764c6d_1 conda-forge aiosignal 1.2.0 pyhd8ed1ab_0 conda-forge alembic 1.8.0 pyhd8ed1ab_0 conda-forge appdirs 1.4.4 pypi_0 pypi arch 5.3.1 pypi_0 pypi argon2-cffi 21.3.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 pypi_0 pypi asttokens 2.0.5 pypi_0 pypi astunparse 1.6.3 pyhd8ed1ab_0 conda-forge async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge attrs 21.4.0 pyhd8ed1ab_0 conda-forge autopage 0.5.1 pyhd8ed1ab_0 conda-forge autots 0.4.2 pyhd8ed1ab_0 conda-forge backcall 0.2.0 pypi_0 pypi backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.11.1 pypi_0 pypi blas 1.0 mkl conda-forge bleach 5.0.0 pypi_0 pypi blinker 1.4 py_1 conda-forge brotli 1.0.9 h166bdaf_7 conda-forge brotli-bin 1.0.9 h166bdaf_7 conda-forge brotlipy 0.7.0 py310h5764c6d_1004 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge ca-certificates 2022.6.15 ha878542_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.0.0 pyhd8ed1ab_0 conda-forge certifi 2022.6.15 py310hff52083_0 conda-forge cffi 1.15.0 py310h0fdd8cc_0 conda-forge charset-normalizer 2.0.12 pyhd8ed1ab_0 conda-forge click 8.1.3 py310hff52083_0 conda-forge cliff 3.10.1 pyhd8ed1ab_0 conda-forge clikit 0.6.2 pypi_0 pypi cloudpickle 2.1.0 pyhd8ed1ab_0 conda-forge cmaes 0.8.2 pyh44b312d_0 conda-forge cmd2 2.3.3 py310hff52083_1 conda-forge cmdstanpy 0.9.5 pypi_0 pypi colorama 0.4.5 pyhd8ed1ab_0 conda-forge colorlog 6.6.0 py310hff52083_1 conda-forge convertdate 2.4.0 pypi_0 pypi crashtest 0.3.1 pypi_0 pypi cryptography 37.0.1 py310h9ce1e76_0
cudatoolkit 11.3.1 h9edb442_10 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge cython 0.29.30 pypi_0 pypi dask 2022.6.1 pypi_0 pypi debugpy 1.6.0 pypi_0 pypi decorator 5.1.1 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi dill 0.3.5.1 pypi_0 pypi distributed 2022.6.1 pypi_0 pypi entrypoints 0.4 pypi_0 pypi ephem 4.1.3 pypi_0 pypi executing 0.8.3 pypi_0 pypi fastjsonschema 2.15.3 pypi_0 pypi fonttools 4.33.3 pypi_0 pypi freetype 2.10.4 h0708190_1 conda-forge frozenlist 1.3.0 py310h5764c6d_1 conda-forge fsspec 2022.5.0 pyhd8ed1ab_0 conda-forge future 0.18.2 py310hff52083_5 conda-forge gast 0.5.3 pyhd8ed1ab_0 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge gluonts 0.10.1 pypi_0 pypi google-auth 2.9.0 pyh6c4a22f_0 conda-forge google-auth-oauthlib 0.4.1 py_2 conda-forge google-pasta 0.2.0 pyh8c360ce_0 conda-forge greenlet 1.1.2 py310hd8f1fbe_2 conda-forge greykite 0.3.0 pypi_0 pypi grpc-cpp 1.45.2 h3b8df00_4 conda-forge grpcio 1.45.0 py310h44b9e0c_0 conda-forge h5py 3.7.0 nompi_py310h06dffec_100 conda-forge hdf5 1.12.1 nompi_h2386368_104 conda-forge heapdict 1.0.1 pypi_0 pypi hijri-converter 2.2.4 pypi_0 pypi holidays 0.14.2 pypi_0 pypi holidays-ext 0.0.7 pypi_0 pypi httpstan 4.7.2 pypi_0 pypi icu 70.1 h27087fc_0 conda-forge idna 3.3 pyhd8ed1ab_0 conda-forge importlib-metadata 4.11.4 py310hff52083_0 conda-forge importlib_resources 5.8.0 pyhd8ed1ab_0 conda-forge intel-openmp 2022.0.1 h06a4308_3633
ipykernel 6.15.0 pypi_0 pypi ipython 8.4.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi jedi 0.18.1 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi joblib 1.1.0 pyhd8ed1ab_0 conda-forge jpeg 9e h166bdaf_2 conda-forge jsonschema 4.6.0 pypi_0 pypi jupyter-client 7.3.4 pypi_0 pypi jupyter-console 6.4.4 pypi_0 pypi jupyter-core 4.10.0 pypi_0 pypi jupyter-http-over-ws 0.0.8 pypi_0 pypi jupyterlab-pygments 0.2.2 pypi_0 pypi jupyterlab-widgets 1.1.1 pypi_0 pypi keras 2.8.0 pyhd8ed1ab_0 conda-forge keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.3 py310hbf28c38_0 conda-forge korean-lunar-calendar 0.2.1 pypi_0 pypi krb5 1.19.3 h3790be6_0 conda-forge lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge lerc 3.0 h9c3ff4c_0 conda-forge libblas 3.9.0 14_linux64_mkl conda-forge libbrotlicommon 1.0.9 h166bdaf_7 conda-forge libbrotlidec 1.0.9 h166bdaf_7 conda-forge libbrotlienc 1.0.9 h166bdaf_7 conda-forge libcblas 3.9.0 14_linux64_mkl conda-forge libcurl 7.83.1 h7bff187_0 conda-forge libdeflate 1.12 h166bdaf_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 12.1.0 h8d9b700_16 conda-forge libgfortran-ng 12.1.0 h69a702a_16 conda-forge libgfortran5 12.1.0 hdcd56e2_16 conda-forge libgomp 12.1.0 h8d9b700_16 conda-forge liblapack 3.9.0 14_linux64_mkl conda-forge libnghttp2 1.47.0 h727a467_0 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libopenblas 0.3.20 pthreads_h78a6416_0 conda-forge libpng 1.6.37 h753d276_3 conda-forge libprotobuf 3.20.1 h6239696_0 conda-forge libssh2 1.10.0 ha56f1ee_2 conda-forge libstdcxx-ng 12.1.0 ha89aaad_16 conda-forge libtiff 4.4.0 hc85c160_1 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libwebp 1.2.2 h3452ae3_0 conda-forge libwebp-base 1.2.2 h7f98852_1 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libzlib 1.2.12 h166bdaf_1 conda-forge lightgbm 3.3.2 py310h122e73d_0 conda-forge locket 1.0.0 pypi_0 pypi lunarcalendar 0.0.9 pypi_0 pypi lz4-c 1.9.3 h9c3ff4c_1 conda-forge mako 1.2.1 pyhd8ed1ab_0 conda-forge markdown 3.3.7 pyhd8ed1ab_0 conda-forge markupsafe 2.1.1 py310h5764c6d_1 conda-forge marshmallow 3.16.0 pypi_0 pypi matplotlib-base 3.5.2 py310h5701ce4_0 conda-forge matplotlib-inline 0.1.3 pypi_0 pypi mistune 0.8.4 pypi_0 pypi mkl 2022.0.1 h06a4308_117
msgpack 1.0.4 pypi_0 pypi multidict 6.0.2 py310h5764c6d_1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mxnet-cu112 1.9.1 pypi_0 pypi nbclient 0.6.4 pypi_0 pypi nbconvert 6.5.0 pypi_0 pypi nbformat 5.4.0 pypi_0 pypi ncurses 6.3 h27087fc_1 conda-forge nest-asyncio 1.5.5 pypi_0 pypi notebook 6.4.12 pypi_0 pypi numpy 1.23.0 py310h53a5b5f_0 conda-forge oauthlib 3.2.0 pyhd8ed1ab_0 conda-forge openjpeg 2.4.0 hb52868f_1 conda-forge openssl 1.1.1q h166bdaf_0 conda-forge opt_einsum 3.3.0 pyhd8ed1ab_1 conda-forge optuna 2.10.1 pyhd8ed1ab_0 conda-forge packaging 21.3 pyhd8ed1ab_0 conda-forge pandas 1.4.3 py310h769672d_0 conda-forge pandocfilters 1.5.0 pypi_0 pypi parso 0.8.3 pypi_0 pypi partd 1.2.0 pypi_0 pypi pastel 0.2.1 pypi_0 pypi patsy 0.5.2 pyhd8ed1ab_0 conda-forge pbr 5.9.0 pyhd8ed1ab_0 conda-forge pexpect 4.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 9.1.1 pypi_0 pypi pip 22.1.2 pyhd8ed1ab_0 conda-forge pmdarima 1.8.5 pypi_0 pypi prettytable 3.3.0 pypi_0 pypi prometheus-client 0.14.1 pypi_0 pypi prompt-toolkit 3.0.29 pypi_0 pypi property-cached 1.6.4 pypi_0 pypi prophet 1.1 pypi_0 pypi protobuf 3.20.1 py310hd8f1fbe_0 conda-forge psutil 5.9.1 pypi_0 pypi pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi pyasn1 0.4.8 py_0 conda-forge pyasn1-modules 0.2.7 py_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pydantic 1.9.1 pypi_0 pypi pydeprecate 0.3.2 pyhd8ed1ab_0 conda-forge pygments 2.12.0 pypi_0 pypi pyjwt 2.4.0 pyhd8ed1ab_0 conda-forge pylev 1.4.0 pypi_0 pypi pymeeus 0.5.11 pypi_0 pypi pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pyperclip 1.8.2 pyhd8ed1ab_2 conda-forge pyrsistent 0.18.1 pypi_0 pypi pysimdjson 3.2.0 pypi_0 pypi pysocks 1.7.1 py310hff52083_5 conda-forge pystan 3.4.0 pypi_0 pypi python 3.10.5 h582c2e5_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-flatbuffers 2.0 pyhd8ed1ab_0 conda-forge python_abi 3.10 2_cp310 conda-forge pytorch 1.12.0 py3.10_cuda11.3_cudnn8.3.2_0 pytorch pytorch-forecasting 0.10.2 pyhd8ed1ab_0 conda-forge pytorch-lightning 1.6.4 pyhd8ed1ab_0 conda-forge pytorch-mutex 1.0 cuda pytorch pytz 2022.1 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pyyaml 5.4.1 pypi_0 pypi pyzmq 23.2.0 pypi_0 pypi qtconsole 5.3.1 pypi_0 pypi qtpy 2.1.0 pypi_0 pypi re2 2022.06.01 h27087fc_0 conda-forge readline 8.1.2 h0f457ee_0 conda-forge requests 2.28.0 pyhd8ed1ab_0 conda-forge requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge rsa 4.8 pyhd8ed1ab_0 conda-forge scikit-learn 1.1.1 py310hffb9edd_0 conda-forge scipy 1.8.1 py310h7612f91_0 conda-forge seaborn 0.11.2 pypi_0 pypi send2trash 1.8.0 pypi_0 pypi setuptools 59.5.0 py310hff52083_0 conda-forge setuptools-git 1.2 pypi_0 pypi six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.9 hbd366e4_1 conda-forge sortedcontainers 2.4.0 pypi_0 pypi soupsieve 2.3.2.post1 pypi_0 pypi sqlalchemy 1.4.39 py310h5764c6d_0 conda-forge sqlite 3.38.5 h4ff8645_0 conda-forge stack-data 0.3.0 pypi_0 pypi statsmodels 0.13.2 py310hde88566_0 conda-forge stevedore 3.5.0 py310hff52083_3 conda-forge tblib 1.7.0 pypi_0 pypi tensorboard 2.8.0 pyhd8ed1ab_1 conda-forge tensorboard-data-server 0.6.0 py310h597c629_2 conda-forge tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge tensorflow 2.8.1 cpu_py310hd1aba9c_0 conda-forge tensorflow-base 2.8.1 cpu_py310h17449b8_0 conda-forge tensorflow-estimator 2.8.1 cpu_py310had6d012_0 conda-forge termcolor 1.1.0 pyhd8ed1ab_3 conda-forge terminado 0.15.0 pypi_0 pypi threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tinycss2 1.1.1 pypi_0 pypi tk 8.6.12 h27826a3_0 conda-forge toolz 0.11.2 pypi_0 pypi torchmetrics 0.9.2 pyhd8ed1ab_0 conda-forge tornado 6.1 pypi_0 pypi tqdm 4.64.0 pyhd8ed1ab_0 conda-forge traitlets 5.3.0 pypi_0 pypi typing-extensions 4.3.0 hd8ed1ab_0 conda-forge typing_extensions 4.3.0 pyha770c72_0 conda-forge tzdata 2022a h191b570_0 conda-forge ujson 5.3.0 pypi_0 pypi unicodedata2 14.0.0 py310h5764c6d_1 conda-forge urllib3 1.26.9 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webargs 8.1.0 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi werkzeug 2.1.2 pyhd8ed1ab_1 conda-forge wheel 0.37.1 pyhd8ed1ab_0 conda-forge widgetsnbextension 3.6.1 pypi_0 pypi wrapt 1.14.1 py310h5764c6d_0 conda-forge xgboost 1.6.1 pypi_0 pypi xlrd 2.0.1 pypi_0 pypi xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h7f98852_2 conda-forge yarl 1.7.2 py310h5764c6d_2 conda-forge zict 2.2.0 pypi_0 pypi zipp 3.8.0 pyhd8ed1ab_0 conda-forge zlib 1.2.12 h166bdaf_1 conda-forge zstd 1.5.2 h8a70e8d_2 conda-forge ``

winedarksea commented 2 years ago

Thanks for good bug report. Almost certainly TensorflowSTS is broken - it wasn't performing very well and I haven't maintained it in two years (note that it is in the 'Experimental' section of the model list). I will put it on my list to check into it.

I can guarantee that GluonTS and pytorch-forecasting are working on the latest release as I use them consistently. model_list = ['GluonTS', 'PytorchForecasting', 'NeuralProphet', 'MultivariateRegression'] models_mode='neuralnets' if you want the `Regression models to only use Tensorflow, although it actually does better with non-neural net models, as is generally the case.

In my experience GluonTS is the best of the lot, so focus on that.

python -m pip install mxnet --no-deps     # check the mxnet documentation for more install options, also try pip install mxnet --no-deps
python -m pip install gluonts