search

automl / Auto-PyTorch

Automatic architecture search and hyperparameter optimization for PyTorch
Apache License 2.0
2.37k stars 287 forks source link

Score function error on example code #484

Open antbz opened 1 year ago

antbz commented 1 year ago

NOTE: ISSUES ARE NOT FOR CODE HELP - Ask for Help at https://stackoverflow.com

Your issue may already be reported! Also, please search on the issue tracker before creating one.

Issue Description

When running the example code provided on the README file, the api.score function throws an exception during an internal call due to missing arguments.

Expected Behavior

A score value to be calculated based on the predictions and actual values.

Current Behavior

A TypeError is thrown due to an internal call in the score function.

Your Code

from autoPyTorch.api.time_series_forecasting import TimeSeriesForecastingTask

# data and metric imports
from sktime.datasets import load_longley
targets, features = load_longley()

# define the forecasting horizon
forecasting_horizon = 3

# Dataset optimized by APT-TS can be a list of np.ndarray/ pd.DataFrame where each series represents an element in the 
# list, or a single pd.DataFrame that records the series
# index information: to which series the timestep belongs? This id can be stored as the DataFrame's index or a separate
# column
# Within each series, we take the last forecasting_horizon as test targets. The items before that as training targets
# Normally the value to be forecasted should follow the training sets
y_train = [targets[: -forecasting_horizon]]
y_test = [targets[-forecasting_horizon:]]

# same for features. For uni-variant models, X_train, X_test can be omitted and set as None
X_train = [features[: -forecasting_horizon]]
# Here x_test indicates the 'known future features': they are the features known previously, features that are unknown
# could be replaced with NAN or zeros (which will not be used by our networks). If no feature is known beforehand,
# we could also omit X_test
known_future_features = list(features.columns)
X_test = [features[-forecasting_horizon:]]

start_times = [targets.index.to_timestamp()[0]]
freq = '1Y'

# initialise Auto-PyTorch api
api = TimeSeriesForecastingTask()

# Search for an ensemble of machine learning algorithms
api.search(
    X_train=X_train,
    y_train=y_train,
    X_test=X_test, 
    optimize_metric='mean_MAPE_forecasting',
    n_prediction_steps=forecasting_horizon,
    memory_limit=16 * 1024,  # Currently, forecasting models use much more memories
    freq=freq,
    start_times=start_times,
    func_eval_time_limit_secs=50,
    total_walltime_limit=60,
    min_num_test_instances=1000,  # proxy validation sets. This only works for the tasks with more than 1000 series
    known_future_features=known_future_features,
)

# our dataset could directly generate sequences for new datasets
test_sets = api.dataset.generate_test_seqs()

# Calculate test accuracy
y_pred = api.predict(test_sets)
score = api.score(y_pred, y_test)
print("Forecasting score", score)

Error message

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-4599e5448863> in <module>
     52 # Calculate test accuracy
     53 y_pred = api.predict(test_sets)
---> 54 score = api.score(y_pred, y_test)
     55 print("Forecasting score", score)

/usr/local/lib/python3.8/dist-packages/autoPyTorch/api/base_task.py in score(self, y_pred, y_test)
   1908             raise ValueError("AutoPytorch failed to infer a task type from the dataset "
   1909                              "Please check the log file for related errors. ")
-> 1910         return calculate_score(target=y_test, prediction=y_pred,
   1911                                task_type=STRING_TO_TASK_TYPES[self.task_type],
   1912                                metrics=[self._metric])

/usr/local/lib/python3.8/dist-packages/autoPyTorch/pipeline/components/training/metrics/utils.py in calculate_score(target, prediction, task_type, metrics, **score_kwargs)
    144                 score_dict[metric_.name] = metric_._sign * metric_(target_scaled, cprediction_scaled, **score_kwargs)
    145             else:
--> 146                 score_dict[metric_.name] = metric_._sign * metric_(target, cprediction, **score_kwargs)
    147     elif task_type in REGRESSION_TASKS:
    148         cprediction = sanitize_array(prediction)

TypeError: __call__() missing 2 required positional arguments: 'sp' and 'n_prediction_steps'

Local environment

pip freeze ``` absl-py==1.3.0 aeppl==0.0.33 aesara==2.7.9 aiohttp==3.8.3 aiosignal==1.3.1 alabaster==0.7.12 albumentations==1.2.1 alembic==1.8.1 altair==4.2.0 appdirs==1.4.4 arviz==0.12.1 astor==0.8.1 astropy==4.3.1 astunparse==1.6.3 async-timeout==4.0.2 asynctest==0.13.0 atari-py==0.2.9 atomicwrites==1.4.1 attrs==22.1.0 audioread==3.0.0 autograd==1.5 autopage==0.5.1 autoPyTorch==0.2.1 Babel==2.11.0 backcall==0.2.0 beautifulsoup4==4.6.3 bleach==5.0.1 blis==0.7.9 bokeh==2.3.3 branca==0.6.0 bs4==0.0.1 CacheControl==0.12.11 cached-property==1.5.2 cachetools==5.2.0 catalogue==2.0.8 catboost==1.1.1 certifi==2022.9.24 cffi==1.15.1 cftime==1.6.2 chardet==3.0.4 charset-normalizer==2.1.1 click==8.1.3 cliff==4.1.0 clikit==0.6.2 cloudpickle==2.2.0 cmaes==0.9.0 cmake==3.22.6 cmd2==2.4.2 cmdstanpy==1.0.8 colorcet==3.0.1 colorlog==6.7.0 colorlover==0.3.0 community==1.0.0b1 confection==0.0.3 ConfigSpace==0.6.0 cons==0.4.5 contextlib2==0.5.5 contourpy==1.0.6 convertdate==2.4.0 crashtest==0.3.1 crcmod==1.7 cufflinks==0.17.3 cupy-cuda11x==11.0.0 cvxopt==1.3.0 cvxpy==1.2.2 cycler==0.11.0 cymem==2.0.7 Cython==0.29.32 daft==0.0.4 dask==2022.12.0 datascience==0.17.5 db-dtypes==1.0.4 debugpy==1.0.0 decorator==4.4.2 defusedxml==0.7.1 Deprecated==1.2.13 descartes==1.1.0 dill==0.3.6 distributed==2022.12.0 dlib==19.24.0 dm-tree==0.1.7 dnspython==2.2.1 docutils==0.17.1 dopamine-rl==1.0.5 earthengine-api==0.1.332 easydict==1.10 ecos==2.0.10 editdistance==0.5.3 emcee==3.1.3 en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.4.1/en_core_web_sm-3.4.1-py3-none-any.whl entrypoints==0.4 ephem==4.1.3 et-xmlfile==1.1.0 etils==0.9.0 etuples==0.3.8 fa2==0.3.5 fastai==2.7.10 fastcore==1.5.27 fastdownload==0.0.7 fastdtw==0.3.4 fastjsonschema==2.16.2 fastprogress==1.0.3 fastrlock==0.8.1 feather-format==0.4.1 filelock==3.8.0 fire==0.4.0 firebase-admin==5.3.0 fix-yahoo-finance==0.0.22 flaky==3.7.0 Flask==1.1.4 flatbuffers==1.12 folium==0.12.1.post1 fonttools==4.38.0 frozenlist==1.3.3 fsspec==2022.11.0 future==0.16.0 gast==0.4.0 GDAL==2.2.2 gdown==4.4.0 gensim==3.6.0 geographiclib==1.52 geopy==1.17.0 gin-config==0.5.0 glob2==0.7 gluonts==0.11.3 google==2.0.3 google-api-core==2.8.2 google-api-python-client==1.12.11 google-auth==2.15.0 google-auth-httplib2==0.0.4 google-auth-oauthlib==0.4.6 google-cloud-bigquery==3.3.6 google-cloud-bigquery-storage==2.16.2 google-cloud-core==2.3.2 google-cloud-datastore==2.9.0 google-cloud-firestore==2.7.2 google-cloud-language==2.6.1 google-cloud-storage==2.5.0 google-cloud-translate==3.8.4 google-colab @ file:///colabtools/dist/google-colab-1.0.0.tar.gz google-crc32c==1.5.0 google-pasta==0.2.0 google-resumable-media==2.4.0 googleapis-common-protos==1.57.0 googledrivedownloader==0.4 graphviz==0.20.1 greenlet==2.0.1 grpcio==1.51.1 grpcio-status==1.48.2 gspread==3.4.2 gspread-dataframe==3.0.8 gym==0.25.2 gym-notices==0.0.8 h5py==3.1.0 HeapDict==1.0.1 hijri-converter==2.2.4 holidays==0.17 holoviews==1.14.9 html5lib==1.0.1 httpimport==0.5.18 httplib2==0.17.4 httpstan==4.6.1 humanize==0.5.1 hyperopt==0.1.2 idna==3.4 imageio==2.22.4 imagesize==1.4.1 imbalanced-learn==0.8.1 imblearn==0.0 imgaug==0.4.0 importlib-metadata==5.1.0 importlib-resources==5.10.0 imutils==0.5.4 inflect==2.1.0 intel-openmp==2022.2.1 intervaltree==2.1.0 ipykernel==5.3.4 ipython==7.9.0 ipython-genutils==0.2.0 ipython-sql==0.3.9 ipywidgets==7.7.1 itsdangerous==1.1.0 jax==0.3.25 jaxlib @ https://storage.googleapis.com/jax-releases/cuda11/jaxlib-0.3.25+cuda11.cudnn805-cp38-cp38-manylinux2014_x86_64.whl jieba==0.42.1 Jinja2==3.1.2 joblib==1.2.0 jpeg4py==0.1.4 jsonschema==4.3.3 jupyter-client==6.1.12 jupyter-console==6.1.0 jupyter-core==4.11.2 jupyterlab-widgets==3.0.3 kaggle==1.5.12 kapre==0.3.7 keras==2.9.0 Keras-Preprocessing==1.1.2 keras-vis==0.4.1 kiwisolver==1.4.4 korean-lunar-calendar==0.3.1 langcodes==3.3.0 libclang==14.0.6 librosa==0.8.1 lightgbm==3.3.3 lightning-utilities==0.3.0 llvmlite==0.39.1 lmdb==0.99 locket==1.0.0 lockfile==0.12.2 logical-unification==0.4.5 LunarCalendar==0.0.9 lxml==4.9.1 Mako==1.2.4 Markdown==3.4.1 MarkupSafe==2.1.1 marshmallow==3.19.0 matplotlib==3.6.2 matplotlib-venn==0.11.7 miniKanren==1.0.3 missingno==0.5.1 mistune==0.8.4 mizani==0.7.3 mkl==2019.0 mlxtend==0.14.0 more-itertools==9.0.0 moviepy==0.2.3.5 mpmath==1.2.1 msgpack==1.0.4 multidict==6.0.3 multipledispatch==0.6.0 multitasking==0.0.11 murmurhash==1.0.9 music21==5.5.0 natsort==5.5.0 nbconvert==5.6.1 nbformat==5.7.0 netCDF4==1.6.2 networkx==2.8.8 nibabel==3.0.2 nltk==3.7 notebook==5.7.16 numba==0.56.4 numexpr==2.8.4 numpy==1.22.4 nvidia-cublas-cu11==11.10.3.66 nvidia-cuda-nvrtc-cu11==11.7.99 nvidia-cuda-runtime-cu11==11.7.99 nvidia-cudnn-cu11==8.5.0.96 oauth2client==4.1.3 oauthlib==3.2.2 okgrade==0.4.3 opencv-contrib-python==4.6.0.66 opencv-python==4.6.0.66 opencv-python-headless==4.6.0.66 openpyxl==3.0.10 opt-einsum==3.3.0 optuna==2.10.1 osqp==0.6.2.post0 packaging==21.3 palettable==3.3.0 pandas==1.5.2 pandas-datareader==0.9.0 pandas-gbq==0.17.9 pandas-profiling==1.4.1 pandocfilters==1.5.0 panel==0.12.1 param==1.12.2 parso==0.8.3 partd==1.3.0 pastel==0.2.1 pathlib==1.0.1 pathy==0.9.0 patsy==0.5.3 pbr==5.11.0 pep517==0.13.0 pexpect==4.8.0 pickleshare==0.7.5 Pillow==9.3.0 pip-tools==6.2.0 plotly==5.11.0 plotnine==0.8.0 pluggy==0.7.1 pooch==1.6.0 portpicker==1.3.9 prefetch-generator==1.0.3 preshed==3.0.8 prettytable==3.5.0 progressbar2==3.38.0 prometheus-client==0.15.0 promise==2.3 prompt-toolkit==2.0.10 prophet==1.1.1 proto-plus==1.22.1 protobuf==3.20.1 psutil==5.9.4 psycopg2==2.9.5 ptyprocess==0.7.0 py==1.11.0 pyarrow==9.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycocotools==2.0.6 pycparser==2.21 pyct==0.4.8 pydantic==1.10.2 pydata-google-auth==1.4.0 pydot==1.3.0 pydot-ng==2.0.0 pydotplus==2.0.2 PyDrive==1.3.1 pyemd==0.5.1 pyerfa==2.0.0.1 Pygments==2.6.1 pygobject==3.26.1 pylev==1.4.0 pymc==4.1.4 PyMeeus==0.5.11 pymongo==4.3.3 pymystem3==0.2.0 pynisher==0.6.4 PyOpenGL==3.1.6 pyparsing==3.0.9 pyperclip==1.8.2 pyrfr==0.8.3 pyrsistent==0.19.2 pysimdjson==3.2.0 pysndfile==1.3.8 PySocks==1.7.1 pystan==3.3.0 pytest==3.6.4 python-apt==0.0.0 python-dateutil==2.8.2 python-louvain==0.16 python-slugify==7.0.0 python-utils==3.4.5 pytorch-forecasting==0.10.3 pytorch-lightning==1.8.3.post1 pytz==2022.6 pyviz-comms==2.2.1 PyWavelets==1.4.1 PyYAML==6.0 pyzmq==23.2.1 qdldl==0.1.5.post2 qudida==0.0.4 regex==2022.10.31 requests==2.28.1 requests-oauthlib==1.3.1 resampy==0.4.2 rpy2==3.5.5 rsa==4.9 scikit-image==0.19.3 scikit-learn==0.24.2 scipy==1.9.3 screen-resolution-extra==0.0.0 scs==3.2.2 seaborn==0.11.2 Send2Trash==1.8.0 setuptools-git==1.2 Shapely==1.8.5.post1 six==1.16.0 sklearn-pandas==1.8.0 sktime==0.14.1 smac==1.4.0 smart-open==5.2.1 snowballstemmer==2.2.0 sortedcontainers==2.4.0 soundfile==0.11.0 spacy==3.4.3 spacy-legacy==3.0.10 spacy-loggers==1.0.3 Sphinx==1.8.6 sphinxcontrib-serializinghtml==1.1.5 sphinxcontrib-websupport==1.2.4 SQLAlchemy==1.4.44 sqlparse==0.4.3 srsly==2.4.5 statsmodels==0.13.5 stevedore==4.1.1 sympy==1.7.1 tables==3.7.0 tabulate==0.9.0 tblib==1.7.0 tenacity==8.1.0 tensorboard==2.11.0 tensorboard-data-server==0.6.1 tensorboard-plugin-wit==1.8.1 tensorboardX==2.5.1 tensorflow==2.9.2 tensorflow-datasets==4.6.0 tensorflow-estimator==2.9.0 tensorflow-gcs-config==2.9.1 tensorflow-hub==0.12.0 tensorflow-io-gcs-filesystem==0.28.0 tensorflow-metadata==1.11.0 tensorflow-probability==0.17.0 termcolor==2.1.1 terminado==0.13.3 testpath==0.6.0 text-unidecode==1.3 textblob==0.15.3 thinc==8.1.5 threadpoolctl==3.1.0 tifffile==2022.10.10 toml==0.10.2 tomli==2.0.1 toolz==0.12.0 torch==1.13.0 torchaudio @ https://download.pytorch.org/whl/cu113/torchaudio-0.12.1%2Bcu113-cp38-cp38-linux_x86_64.whl torchmetrics==0.11.0 torchsummary==1.5.1 torchtext==0.13.1 torchvision==0.14.0 tornado==6.2 tqdm==4.64.1 traitlets==5.1.1 tweepy==3.10.0 typeguard==2.7.1 typer==0.7.0 typing-extensions==4.4.0 tzlocal==1.5.1 uritemplate==3.0.1 urllib3==1.26.13 vega-datasets==0.9.0 wasabi==0.10.1 wcwidth==0.2.5 webargs==8.2.0 webencodings==0.5.1 Werkzeug==2.2.2 widgetsnbextension==3.6.1 wordcloud==1.8.2.2 wrapt==1.14.1 xarray==0.20.2 xarray-einstats==0.2.2 xgboost==0.90 xkit==0.0.0 xlrd==1.1.0 xlwt==1.3.0 yarl==1.8.2 yellowbrick==1.5 zict==2.2.0 zipp==3.11.0 ```
dengdifan commented 1 year ago

Hi, Thanks for the report. We are currently working on this issue and will solve that in the next release.

ericleonardo commented 1 year ago

Please, are you still able to run Auto-PyTorch on Google Colab? I'm having trouble during installation on Colab. Tried many ways but doesn't install. How did you installed it? Thanks @antbz