search

quantopian / zipline

Zipline, a Pythonic Algorithmic Trading Library
https://www.zipline.io
Apache License 2.0
17.27k stars 4.67k forks source link

Using minute data and csvdir, data.history crashes when asking for daily history #2385

Open brettelliot opened 5 years ago

brettelliot commented 5 years ago

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

* Operating System: MacOS Mojave * Python Version: 3.5.6 * How did you install Zipline: mac ports * Python packages: alembic==1.0.3 appnope==0.1.0 backcall==0.1.0 bcolz==0.12.1 bleach==3.0.2 Bottleneck==1.2.1 certifi==2018.10.15 chardet==3.0.4 Click==7.0 contextlib2==0.5.5 cycler==0.10.0 cyordereddict==1.0.0 Cython==0.29.1 decorator==4.3.0 defusedxml==0.5.0 empyrical==0.5.0 entrypoints==0.2.3 idna==2.7 intervaltree==2.1.0 ipykernel==5.1.0 ipython==7.1.1 ipython-genutils==0.2.0 ipywidgets==7.4.2 jedi==0.13.1 Jinja2==2.10 jsonschema==2.6.0 jupyter==1.0.0 jupyter-client==5.2.3 jupyter-console==6.0.0 jupyter-core==4.4.0 kiwisolver==1.0.1 Logbook==1.4.1 lru-dict==1.1.6 lxml==4.2.5 Mako==1.0.7 MarkupSafe==1.1.0 matplotlib==3.0.2 mistune==0.8.4 multipledispatch==0.6.0 nbconvert==5.4.0 nbformat==4.4.0 networkx==1.11 notebook==5.7.2 numexpr==2.6.8 numpy==1.15.4 pandas==0.22.0 pandas-datareader==0.7.0 pandocfilters==1.4.2 parso==0.3.1 patsy==0.5.1 pexpect==4.6.0 pickleshare==0.7.5 Pillow==5.3.0 prometheus-client==0.4.2 prompt-toolkit==2.0.7 ptyprocess==0.6.0 pyEX==0.1.5 Pygments==2.3.0 pyparsing==2.3.0 python-dateutil==2.7.5 python-editor==1.0.3 pytz==2018.7 PyYAML==3.13 pyzmq==17.1.2 qtconsole==4.4.3 requests==2.20.1 requests-file==1.4.3 scipy==1.1.0 Send2Trash==1.5.0 six==1.11.0 socketIO-client-nexus==0.7.6 sortedcontainers==2.1.0 SQLAlchemy==1.2.14 statsmodels==0.9.0 tables==3.4.4 terminado==0.8.1 testpath==0.4.2 toolz==0.9.0 tornado==5.1.1 trading-calendars==1.5.1 traitlets==4.3.2 urllib3==1.24.1 wcwidth==0.1.7 webencodings==0.5.1 websocket-client==0.54.0 widgetsnbextension==3.4.2 wrapt==1.10.11 zipline==1.3.0

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

Here is how you can reproduce this issue on your machine:

Reproduction Steps

  1. Create a directory structure that has some minute data from IEX. This data is attached here: iex2.zip

./iex ./iex/minute ./iex/minute/AAPL.csv

  1. Put these lines into ~/.zipline/extension.py:
from zipline.data.bundles import register, csvdir
register('my-iex-csvdir', csvdir.csvdir_equities(['minute']))

  1. Ingest: CSVDIR=./iex/ zipline ingest -b my-iex-csvdir

  2. Create a simple algorithm that just calls data.history. Mine's called e0011g.py (rename from .txt)

e0011g.py.txt

  1. Run zipline:

zipline run -f e0011g.py -s '2018-11-01' -e '2018-11-30' -b my-iex-csvdir --data-frequency minute -o /dev/null

  1. Watch it crash instead of printing the daily data. ;)

...

What steps have you taken to resolve this already?

I've tried running the algorithm with different start dates so that there's several days of data before the algo runs.

I've searched github and have seen this issue was fixed twice already (so I know it's a supported feature)

https://github.com/quantopian/zipline/issues/1629 https://github.com/quantopian/zipline/issues/1356

...

Anything else?

Here's a complete stack trace:

Traceback (most recent call last):
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/bin/zipline", line 11, in <module>
    sys.exit(main())
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/__main__.py", line 107, in _
    return f(*args, **kwargs)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/__main__.py", line 276, in run
    blotter=blotter,
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/utils/run_algo.py", line 229, in _run
    overwrite_sim_params=False,
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/algorithm.py", line 756, in run
    for perf in self.get_generator():
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/gens/tradesimulation.py", line 206, in transform
    for capital_change_packet in every_bar(dt):
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/gens/tradesimulation.py", line 134, in every_bar
    handle_data(algo, current_data, dt_to_use)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/utils/events.py", line 216, in handle_data
    dt,
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/utils/events.py", line 235, in handle_data
    self.callback(context, data)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/algorithm.py", line 469, in handle_data
    self._handle_data(self, data)
  File "e0011g.py", line 24, in handle_data
    daily_history = data.history(context.asset, 'price', bar_count=5, frequency="1d")
  File "zipline/_protocol.pyx", line 121, in zipline._protocol.check_parameters.__call__.assert_keywords_and_call
  File "zipline/_protocol.pyx", line 647, in zipline._protocol.BarData.history
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/data/data_portal.py", line 964, in get_history_window
    "close", data_frequency)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/data/data_portal.py", line 806, in _get_history_daily_window
    assets, days_for_window, end_dt, field_to_use, data_frequency
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/data/data_portal.py", line 836, in _get_history_daily_window_data
    days_for_window[0:-1]
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/data/data_portal.py", line 1117, in _get_daily_window_data
    extra_slot)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/data/history_loader.py", line 549, in history
    is_perspective_after)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/data/history_loader.py", line 431, in _ensure_sliding_windows
    array = self._array(prefetch_dts, needed_assets, field)
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/data/history_loader.py", line 573, in _array
    assets,
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/data/dispatch_bar_reader.py", line 120, in load_raw_arrays
    for t in asset_types if sid_groups[t]}
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/data/dispatch_bar_reader.py", line 120, in <dictcomp>
    for t in asset_types if sid_groups[t]}
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/data/us_equity_pricing.py", line 629, in load_raw_arrays
    assets,
  File "/Users/brettelliot/.virtualenvs/maroma-e0011/lib/python3.5/site-packages/zipline/data/us_equity_pricing.py", line 619, in _compute_slices
    assets,
  File "zipline/data/_equities.pyx", line 44, in zipline.data._equities._compute_row_slices (zipline/data/_equities.c:2495)
  File "zipline/data/_equities.pyx", line 103, in zipline.data._equities._compute_row_slices (zipline/data/_equities.c:2189)
KeyError: 0

...

Sincerely, $ whoami

brettelliot commented 5 years ago

Hi zipline devs. Is this a bug you think you'll get to soon-ish or is it low on the priority list? Thanks!

niranjan92 commented 5 years ago

Hi Team, any update on this issue?

erichuang2013 commented 5 years ago

got same problem.

CapitalZe commented 4 years ago

Same here

akamick86 commented 3 years ago

Very similar issue. I can execute daily_history = data.history(context.asset, 'price', bar_count=**1**, frequency="1d") But if I do daily_history = data.history(context.asset, 'price', bar_count=**10**, frequency="1d") #any number but 1 fails. I get the following exception

File "zipline/data/_equities.pyx", line 140, in zipline.data._equities._compute_row_slices ValueError: At least one valid asset id is required.

ymao1993 commented 3 years ago

Same here. Feeling bad that this issue is still not fixed.

Is zipline still an actively maintained repo?