search

quantopian / zipline

Zipline, a Pythonic Algorithmic Trading Library
https://www.zipline.io
Apache License 2.0
17.44k stars 4.71k forks source link

Issue with running strategies based on trading_calendars #2518

Open erykml opened 5 years ago

erykml commented 5 years ago

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

* Operating System: MacOS Mojave 10.14.5 * Python Version: Python 3.5.6 :: Anaconda, Inc. * Python Bitness: 64 * How did you install Zipline: `conda` * Python packages: alembic==0.7.7 appnope==0.1.0 asn1crypto==0.24.0 backcall==0.1.0 bcolz==0.12.1 beautifulsoup4==4.7.1 bleach==2.1.4 Bottleneck==1.2.1 certifi==2018.8.24 cffi==1.11.5 chardet==3.0.4 click==6.7 contextlib2==0.5.5 cryptography==2.3.1 cycler==0.10.0 cyordereddict==0.2.2 Cython==0.28.5 decorator==4.4.0 defusedxml==0.6.0 empyrical==0.5.0 entrypoints==0.2.3 html5lib==1.0.1 idna==2.7 intervaltree==2.1.0 ipykernel==4.10.0 ipython==6.5.0 ipython-genutils==0.2.0 ipywidgets==7.4.1 jedi==0.12.1 Jinja2==2.10 jsonschema==2.6.0 jupyter==1.0.0 jupyter-client==5.3.1 jupyter-console==5.2.0 jupyter-core==4.5.0 kiwisolver==1.0.1 Logbook==0.12.5 lru-dict==1.1.4 lxml==4.2.5 Mako==1.0.10 MarkupSafe==1.0 matplotlib==3.0.0 mistune==0.8.3 mkl-fft==1.0.6 mkl-random==1.0.1 multipledispatch==0.6.0 nbconvert==5.5.0 nbformat==4.4.0 networkx==1.11 notebook==5.6.0 numexpr==2.6.1 numpy==1.14.6 pandas==0.22.0 pandas-datareader==0.6.0 pandocfilters==1.4.2 parso==0.5.0 patsy==0.5.0 pexpect==4.6.0 pickleshare==0.7.4 prometheus-client==0.7.1 prompt-toolkit==1.0.15 ptyprocess==0.6.0 pycparser==2.19 Pygments==2.4.2 pyOpenSSL==18.0.0 pyparsing==2.4.0 PySocks==1.6.8 python-dateutil==2.7.3 pytz==2019.1 pyzmq==17.1.2 qtconsole==4.5.1 requests==2.19.1 requests-file==1.4.3 requests-ftp==0.3.1 scipy==1.1.0 Send2Trash==1.5.0 simplegeneric==0.8.1 six==1.11.0 sortedcontainers==2.0.5 soupsieve==1.9.2 SQLAlchemy==1.2.11 statsmodels==0.9.0 TA-Lib==0.4.17 tables==3.4.4 terminado==0.8.1 testpath==0.3.1 toolz==0.10.0 tornado==5.1.1 trading-calendars==1.8.1 traitlets==4.3.2 urllib3==1.23 watermark==1.8.1 wcwidth==0.1.7 webencodings==0.5.1 widgetsnbextension==3.4.1 wrapt==1.10.11 yahoofinancials==1.5 zipline==1.3.0

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 1483315200000000000

During handling of the above exception, another exception occurred:

KeyError Traceback (most recent call last)

in () ----> 1 get_ipython().run_cell_magic('zipline', '--start 2017-1-2 --end 2019-6-28 --capital-base 1050.0 --bundle eu_stocks -o out.pkl', "\ndef initialize(context):\n set_benchmark('aex')\n context.asset = symbol('abn')\n\ndef handle_data(context, data):\n order_target_percent(context.asset, 0.5)") ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell) 2165 magic_arg_s = self.var_expand(line, stack_depth) 2166 with self.builtin_trap: -> 2167 result = fn(magic_arg_s, cell) 2168 return result 2169 ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/zipline/__main__.py in zipline_magic(line, cell) 309 '%s%%zipline' % ((cell or '') and '%'), 310 # don't use system exit and propogate errors to the caller --> 311 standalone_mode=False, 312 ) 313 except SystemExit as e: ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/click/core.py in main(self, args, prog_name, complete_var, standalone_mode, **extra) 695 try: 696 with self.make_context(prog_name, args, **extra) as ctx: --> 697 rv = self.invoke(ctx) 698 if not standalone_mode: 699 return rv ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/click/core.py in invoke(self, ctx) 893 """ 894 if self.callback is not None: --> 895 return ctx.invoke(self.callback, **ctx.params) 896 897 ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/click/core.py in invoke(*args, **kwargs) 533 with augment_usage_errors(self): 534 with ctx: --> 535 return callback(*args, **kwargs) 536 537 def forward(*args, **kwargs): ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/click/decorators.py in new_func(*args, **kwargs) 15 """ 16 def new_func(*args, **kwargs): ---> 17 return f(get_current_context(), *args, **kwargs) 18 return update_wrapper(new_func, f) 19 ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/zipline/__main__.py in run(ctx, algofile, algotext, define, data_frequency, capital_base, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, blotter) 274 local_namespace=local_namespace, 275 environ=os.environ, --> 276 blotter=blotter, 277 ) 278 ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/zipline/utils/run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter) 167 equity_minute_reader=bundle_data.equity_minute_bar_reader, 168 equity_daily_reader=bundle_data.equity_daily_bar_reader, --> 169 adjustment_reader=bundle_data.adjustment_reader, 170 ) 171 ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/zipline/data/data_portal.py in __init__(self, asset_finder, trading_calendar, first_trading_day, equity_daily_reader, equity_minute_reader, future_daily_reader, future_minute_reader, adjustment_reader, last_available_session, last_available_minute, minute_history_prefetch_length, daily_history_prefetch_length) 289 self._first_trading_day 290 ) --> 291 if self._first_trading_day is not None else (None, None) 292 ) 293 ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/trading_calendars/trading_calendar.py in open_and_close_for_session(self, session_label) 758 # http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#datetime-with-tz # noqa 759 return ( --> 760 sched.at[session_label, 'market_open'].tz_localize(UTC), 761 sched.at[session_label, 'market_close'].tz_localize(UTC), 762 ) ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/pandas/core/indexing.py in __getitem__(self, key) 1867 1868 key = self._convert_key(key) -> 1869 return self.obj._get_value(*key, takeable=self._takeable) 1870 1871 def __setitem__(self, key, value): ~/anaconda3/envs/env_zipline2/lib/python3.5/site-packages/pandas/core/frame.py in _get_value(self, index, col, takeable) 1983 1984 try: -> 1985 return engine.get_value(series._values, index) 1986 except (TypeError, ValueError): 1987 pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value() pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value() pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc() KeyError: Timestamp('2017-01-02 00:00:00+0000', tz='UTC') ``` Here is how you can reproduce this issue on your machine: ## Reproduction Steps 1. I downloaded data into CSV files (two files names `abn.csv` and `aex.csv`). The head of one looks like this: ``` date,open,high,low,close,volume,dividend,split 2017-01-02,21.100000381469727,21.295000076293945,20.94499969482422,21.274999618530273,407919,0,0 2017-01-03,21.334999084472656,22.06999969482422,21.334999084472656,21.950000762939453,1453872,0,0 2017-01-04,21.989999771118164,22.135000228881836,21.934999465942383,22.125,1169976,0,0 2017-01-05,22.114999771118164,22.280000686645508,22.0,22.125,1102774,0,0 2017-01-06,22.100000381469727,22.25,22.045000076293945,22.229999542236328,766804,0,0 2017-01-09,22.21500015258789,22.219999313354492,21.93000030517578,21.989999771118164,1006061,0,0 2017-01-10,22.024999618530273,22.165000915527344,21.875,22.079999923706055,954217,0,0 2017-01-11,22.100000381469727,22.25,21.950000762939453,22.049999237060547,1140509,0,0 2017-01-12,21.90999984741211,22.104999542236328,21.829999923706055,21.850000381469727,1231540,0,0 ``` 2. I modified the `extension.py` file to contain: ``` import pandas as pd from zipline.data.bundles import register from zipline.data.bundles.csvdir import csvdir_equities start_session = pd.Timestamp('2017-01-02', tz='utc') end_session = pd.Timestamp('2019-06-28', tz='utc') # register the bundle register( 'eu_stocks', csvdir_equities( ['daily'], '/path/', ), calendar_name='XAMS', # Euronext Amsterdam start_session=start_session, end_session=end_session ) ``` 3. I ingested the data 4. I ran the following strategy ``` def initialize(context): set_benchmark('aex') context.asset = symbol('abn') def handle_data(context, data): order_target_percent(context.asset, 0.5) ``` ## What steps have you taken to resolve this already? I tried following this [issue](https://github.com/quantopian/zipline/issues/2018) but with no success, as I already have the latest version. Sincerely, erykml
trevor-richardson commented 4 years ago

@erykml I have the exact same issue and I am trying to work through your example medium post. How did you fix this issue? Is it something with the pandas version? I'm using conda on fresh ubuntu 18.04. I'm getting this error walking through -- https://towardsdatascience.com/backtesting-trading-strategies-using-custom-data-in-zipline-e6fd65eeaca0. Thanks!

erykml commented 4 years ago

Hi @trevor-richardson, did you try following all the steps described in this Notebook, while naturally adjusting the paths?