search

awslabs / gluonts

Probabilistic time series modeling in Python
https://ts.gluon.ai
Apache License 2.0
4.62k stars 753 forks source link

Problem when calculating OWA by Evaluator(DeepAR) #800

Closed KanWu10 closed 4 years ago

KanWu10 commented 4 years ago

Hi I got an issue in when I tried to calculate evaluation metrics. In class Evaluator, parameter "calculate_owa" is False defaulted. I set it to True when calling the Evaluator but got an error:"init() got an unexpected keyword argument 'calculate_owa' ".

Screen Shot 2020-04-30 at 10 44 46 AM

Does anybody know how to fix it ?

lostella commented 4 years ago

@KanWu10 what version of GluonTS are you using? That feature is only in the master branch, not in the latest release (0.4.3)

KanWu10 commented 4 years ago

@lostella The version I am using is 0.4.3, how should I update it? thank you

lostella commented 4 years ago

You can do

pip install git+https://github.com/awslabs/gluon-ts.git

The next release (0.5) is expected pretty soon, watch out for it, so in a few days you should be able to install gluonts again from PyPI (the simpler pip install gluonts)

KanWu10 commented 4 years ago

@lostella Thank you! I just tried the command you suggested but got an new error:

Screen Shot 2020-04-30 at 11 01 49 AM

I have no idea of how to deal with it.

lostella commented 4 years ago

could you please copy paste the entire error? use triple-ticks (```) to enclose code snippets so that they're nicely formatted 

like this
KanWu10 commented 4 years ago

Oh sorry about that. I pasted the entire error below.

ERROR: Command errored out with exit status 1:
   command: /Users/dongyuewu/PycharmProjects/KanWu/venv/bin/python /Users/dongyuewu/PycharmProjects/KanWu/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/h8/mmsvg1mx78s2p44h58z7b82m0000gn/T/tmpdd0knsj_
       cwd: /private/var/folders/h8/mmsvg1mx78s2p44h58z7b82m0000gn/T/pip-req-build-gvcynaab
  Complete output (10 lines):
  Traceback (most recent call last):
    File "/Users/dongyuewu/PycharmProjects/KanWu/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/Users/dongyuewu/PycharmProjects/KanWu/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/dongyuewu/PycharmProjects/KanWu/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 108, in get_requires_for_build_wheel
      backend = _build_backend()
    File "/Users/dongyuewu/PycharmProjects/KanWu/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 99, in _build_backend
      obj = getattr(obj, path_part)
  AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/dongyuewu/PycharmProjects/KanWu/venv/bin/python /Users/dongyuewu/PycharmProjects/KanWu/venv/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /var/folders/h8/mmsvg1mx78s2p44h58z7b82m0000gn/T/tmpdd0knsj_ Check the logs for full command output.
lostella commented 4 years ago

Maybe this is related? https://github.com/pypa/setuptools/issues/1694

KanWu10 commented 4 years ago

@lostella Thank you ! I will try this.

KanWu10 commented 4 years ago

@lostella I fixed the last error and my current package version is gluonts-0.4.3.dev117+g53120d4. However I still cannot set feature 'calculate_owa' for the same error.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-85-78af253b4ad1> in <module>
----> 1 agg_metrics, item_metrics = Evaluator(calculate_owa=True)(
      2         ts_it, forecast_it, num_series=len(test_data)
      3     )

TypeError: __init__() got an unexpected keyword argument 'calculate_owa'
lostella commented 4 years ago

Now that’s weird. Can you check what the following outputs?

import gluonts
print(gluonts.__version__)
from gluonts.evaluation import Evaluator
ev = Evaluator(calculate_owa=True)
KanWu10 commented 4 years ago

@lostella Hi below is the output.

Screen Shot 2020-04-30 at 2 03 23 PM
lostella commented 4 years ago

So you're still running 0.4.3. Can you pip uninstall, make sure the package is removed, and reinstall it with the command I pasted above?

KanWu10 commented 4 years ago

@lostella I have removed the gluonts 0.4.3 before I use the command to reinstall and the package version in Preference is also gluonts-0.4.3.dev117+g53120d4.

(venv) KanWuMacBookPro:KanWu dongyuewu$ pip install --no-use-pep517  git+https://github.com/awslabs/gluon-ts.git
Collecting git+https://github.com/awslabs/gluon-ts.git
  Cloning https://github.com/awslabs/gluon-ts.git to /private/var/folders/h8/mmsvg1mx78s2p44h58z7b82m0000gn/T/pip-req-build-ycig9zja
  Running command git clone -q https://github.com/awslabs/gluon-ts.git /private/var/folders/h8/mmsvg1mx78s2p44h58z7b82m0000gn/T/pip-req-build-ycig9zja
Requirement already satisfied: holidays<0.10,>=0.9 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (0.9.12)
Requirement already satisfied: matplotlib~=3.0 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (3.2.1)
Requirement already satisfied: numpy~=1.16 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (1.18.2)
Collecting pandas~=1.0
  Using cached pandas-1.0.3-cp37-cp37m-macosx_10_9_x86_64.whl (10.0 MB)
Requirement already satisfied: pydantic~=1.1 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (1.5.1)
Requirement already satisfied: tqdm~=4.23 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (4.45.0)
Requirement already satisfied: ujson~=1.35 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (1.35)
Requirement already satisfied: python-dateutil in ./venv/lib/python3.7/site-packages (from holidays<0.10,>=0.9->gluonts==0.4.3.dev117+g53120d4) (2.8.0)
Requirement already satisfied: six in ./venv/lib/python3.7/site-packages (from holidays<0.10,>=0.9->gluonts==0.4.3.dev117+g53120d4) (1.14.0)
Requirement already satisfied: cycler>=0.10 in ./venv/lib/python3.7/site-packages (from matplotlib~=3.0->gluonts==0.4.3.dev117+g53120d4) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in ./venv/lib/python3.7/site-packages (from matplotlib~=3.0->gluonts==0.4.3.dev117+g53120d4) (2.4.7)
Requirement already satisfied: kiwisolver>=1.0.1 in ./venv/lib/python3.7/site-packages (from matplotlib~=3.0->gluonts==0.4.3.dev117+g53120d4) (1.2.0)
Requirement already satisfied: pytz>=2017.2 in ./venv/lib/python3.7/site-packages (from pandas~=1.0->gluonts==0.4.3.dev117+g53120d4) (2019.3)
Building wheels for collected packages: gluonts
  Building wheel for gluonts (setup.py) ... done
  Created wheel for gluonts: filename=gluonts-0.4.3.dev117+g53120d4-py3-none-any.whl size=418103 sha256=53e6845d3722495052f02d39e2daa20ed66722b41c4e2d19fb6e9808555d53cc
  Stored in directory: /private/var/folders/h8/mmsvg1mx78s2p44h58z7b82m0000gn/T/pip-ephem-wheel-cache-8m4lmqim/wheels/f5/db/5e/07695b2f0357956b0f6c0d36c43a7c054a0b6647e6bbc19566
Successfully built gluonts
Installing collected packages: pandas, gluonts
  Attempting uninstall: pandas
    Found existing installation: pandas 0.25.3
    Uninstalling pandas-0.25.3:
      Successfully uninstalled pandas-0.25.3
  Attempting uninstall: gluonts
    Found existing installation: gluonts 0.4.3
    Uninstalling gluonts-0.4.3:
      Successfully uninstalled gluonts-0.4.3
Successfully installed gluonts-0.4.3.dev117+g53120d4 pandas-1.0.3
(venv) KanWuMacBookPro:KanWu dongyuewu$ pip install --no-use-pep517  git+https://github.com/awslabs/gluon-ts.git
Collecting git+https://github.com/awslabs/gluon-ts.git
  Cloning https://github.com/awslabs/gluon-ts.git to /private/var/folders/h8/mmsvg1mx78s2p44h58z7b82m0000gn/T/pip-req-build-o18xcpbz
  Running command git clone -q https://github.com/awslabs/gluon-ts.git /private/var/folders/h8/mmsvg1mx78s2p44h58z7b82m0000gn/T/pip-req-build-o18xcpbz
Requirement already satisfied: holidays<0.10,>=0.9 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (0.9.12)
Requirement already satisfied: matplotlib~=3.0 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (3.2.1)
Requirement already satisfied: numpy~=1.16 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (1.18.2)
Requirement already satisfied: pandas~=1.0 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (1.0.3)
Requirement already satisfied: pydantic~=1.1 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (1.5.1)
Requirement already satisfied: tqdm~=4.23 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (4.45.0)
Requirement already satisfied: ujson~=1.35 in ./venv/lib/python3.7/site-packages (from gluonts==0.4.3.dev117+g53120d4) (1.35)
Requirement already satisfied: python-dateutil in ./venv/lib/python3.7/site-packages (from holidays<0.10,>=0.9->gluonts==0.4.3.dev117+g53120d4) (2.8.0)
Requirement already satisfied: six in ./venv/lib/python3.7/site-packages (from holidays<0.10,>=0.9->gluonts==0.4.3.dev117+g53120d4) (1.14.0)
Requirement already satisfied: cycler>=0.10 in ./venv/lib/python3.7/site-packages (from matplotlib~=3.0->gluonts==0.4.3.dev117+g53120d4) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in ./venv/lib/python3.7/site-packages (from matplotlib~=3.0->gluonts==0.4.3.dev117+g53120d4) (1.2.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in ./venv/lib/python3.7/site-packages (from matplotlib~=3.0->gluonts==0.4.3.dev117+g53120d4) (2.4.7)
Requirement already satisfied: pytz>=2017.2 in ./venv/lib/python3.7/site-packages (from pandas~=1.0->gluonts==0.4.3.dev117+g53120d4) (2019.3)
Building wheels for collected packages: gluonts
  Building wheel for gluonts (setup.py) ... done
  Created wheel for gluonts: filename=gluonts-0.4.3.dev117+g53120d4-py3-none-any.whl size=418103 sha256=d40588fa827826294d0783f876f51972ef5429ed3ec120c17f7cf2ffcdfc06b5
  Stored in directory: /private/var/folders/h8/mmsvg1mx78s2p44h58z7b82m0000gn/T/pip-ephem-wheel-cache-944jgdch/wheels/f5/db/5e/07695b2f0357956b0f6c0d36c43a7c054a0b6647e6bbc19566
Successfully built gluonts
Installing collected packages: gluonts
Successfully installed gluonts-0.4.3.dev117+g53120d4
Screen Shot 2020-04-30 at 2 57 02 PM

However, when I ran Evaluator command I still got same error.

lostella commented 4 years ago

@KanWu10 did you restart the IPython kernel in your Jupyter notebook after installing gluonts-0.4.3.dev117+g53120d4? The notebook clearly output "0.4.3" above

KanWu10 commented 4 years ago

@lostella My bad. I just restarted it and all stuff are working well now. Thank you so much for your help, I really appreciate it.