search

quantopian / zipline

Zipline, a Pythonic Algorithmic Trading Library
https://www.zipline.io
Apache License 2.0
17.52k stars 4.71k forks source link

Zipline does not work with 1 minute data #1268

Closed nalon99 closed 6 years ago

nalon99 commented 8 years ago

Environment

alabaster                 0.7.7                    py27_0    defaults
anaconda                  4.0.0               np110py27_0    defaults
anaconda-client           1.4.0                    py27_0    defaults
anaconda-navigator        1.1.0                    py27_0    defaults
argcomplete               1.0.0                    py27_1    defaults
astropy                   1.1.2               np110py27_0    defaults
awscli                    1.6.2                     <pip>
babel                     2.2.0                    py27_0    defaults
backports-abc             0.4                       <pip>
backports.ssl-match-hostname 3.5.0.1                   <pip>
backports_abc             0.4                      py27_0    defaults
bcdoc                     0.12.2                    <pip>
bcolz                     0.12.1              np110py27_0    Quantopian
beautifulsoup4            4.4.1                    py27_0    defaults
bitarray                  0.8.1                    py27_1    defaults
blaze                     0.9.1                    py27_0    defaults
bokeh                     0.11.1                   py27_0    defaults
boto                      2.39.0                   py27_0    defaults
botocore                  0.73.0                    <pip>
bottleneck                1.0.0               np110py27_0    defaults
bzip2                     1.0.6                     vc9_2  [vc9]  defaults
cachetools                1.1.6                    py27_0    Quantopian
cdecimal                  2.3                      py27_1    defaults
certifi                   2015.4.28                 <pip>
cffi                      1.5.2                    py27_0    defaults
chest                     0.2.3                    py27_0    defaults
click                     6.6                      py27_0    defaults
cloudpickle               0.1.1                    py27_0    defaults
clyent                    1.2.1                    py27_0    defaults
colorama                  0.3.7                    py27_0    defaults
comtypes                  1.1.2                    py27_0    defaults
conda                     4.0.6                    py27_0    defaults
conda-build               1.20.0                   py27_0    defaults
conda-env                 2.4.5                    py27_0    defaults
conda-manager             0.3.1                    py27_0    defaults
configobj                 5.0.6                    py27_0    defaults
console_shortcut          0.1.1                    py27_1    defaults
contextlib2               0.4.0                    py27_0    defaults
cryptography              1.3                      py27_0    defaults
curl                      7.45.0                    vc9_1  [vc9]  defaults
cycler                    0.10.0                   py27_0    defaults
cyordereddict             0.2.2                    py27_0    Quantopian
cython                    0.23.4                   py27_0    defaults
cytoolz                   0.7.5                    py27_0    defaults
dask                      0.8.1                    py27_0    defaults
datashape                 0.5.1                    py27_0    defaults
decorator                 4.0.9                    py27_0    defaults
dill                      0.2.4                    py27_0    defaults
docutils                  0.12                     py27_1    defaults
enum34                    1.1.2                    py27_0    defaults
et-xmlfile                1.0.1                     <pip>
et_xmlfile                1.0.1                    py27_0    defaults
fastcache                 1.0.2                    py27_0    defaults
flask                     0.10.1                   py27_1    defaults
flask-cors                2.1.2                    py27_0    defaults
freetype                  2.5.5                     vc9_0  [vc9]  defaults
funcsigs                  0.4                      py27_0    defaults
functools32               3.2.3.post2               <pip>
futures                   3.0.3                    py27_0    defaults
genson                    0.1.0                     <pip>
gevent                    1.1.0                    py27_0    defaults
graphlab-create           1.9                       <pip>
graphlab-create-license   1.8.5                     <pip>
greenlet                  0.4.9                    py27_0    defaults
grin                      1.2.1                    py27_2    defaults
h5py                      2.5.0               np110py27_4    defaults
hdf5                      1.8.15.1                  vc9_4  [vc9]  defaults
heapdict                  1.0.0                    py27_0    defaults
idna                      2.0                      py27_0    defaults
inflection                0.3.1                     <pip>
intervaltree              2.1.0                    py27_0    Quantopian
ipaddress                 1.0.14                   py27_0    defaults
ipykernel                 4.3.1                    py27_0    defaults
ipython                   4.1.2                    py27_1    defaults
ipython-genutils          0.1.0                     <pip>
ipython_genutils          0.1.0                    py27_0    defaults
ipywidgets                4.1.1                    py27_0    defaults
itsdangerous              0.24                     py27_0    defaults
jdcal                     1.2                      py27_0    defaults
jedi                      0.9.0                    py27_0    defaults
jinja2                    2.8                      py27_0    defaults
jmespath                  0.5.0                     <pip>
jpeg                      8d                        vc9_0  [vc9]  defaults
jsonschema                2.4.0                    py27_0    defaults
jupyter                   1.0.0                    py27_2    defaults
jupyter-client            4.2.2                     <pip>
jupyter-console           4.1.1                     <pip>
jupyter-core              4.1.0                     <pip>
jupyter_client            4.2.2                    py27_0    defaults
jupyter_console           4.1.1                    py27_0    defaults
jupyter_core              4.1.0                    py27_0    defaults
libpng                    1.6.17                    vc9_1  [vc9]  defaults
librato-metrics           0.4.9                     <pip>
libsodium                 1.0.3                         0    defaults
libtiff                   4.0.6                     vc9_1  [vc9]  defaults
llvmlite                  0.9.0                    py27_0    defaults
locket                    0.2.0                    py27_0    defaults
logbook                   0.12.5                   py27_0    Quantopian
lxml                      3.6.0                    py27_0    defaults
markupsafe                0.23                     py27_1    defaults
matplotlib                1.5.1               np110py27_0    defaults
menuinst                  1.3.2                    py27_0    defaults
mistune                   0.7.2                    py27_0    defaults
mixpanel-py               3.1.1                     <pip>
mkl                       11.3.1                        0    defaults
mkl-service               1.1.2                    py27_0    defaults
more-itertools            2.2                       <pip>
mpmath                    0.19                     py27_0    defaults
multipledispatch          0.4.8                    py27_0    defaults
nbconvert                 4.1.0                    py27_0    defaults
nbformat                  4.0.1                    py27_0    defaults
ndg-httpsclient           0.4.0                     <pip>
networkx                  1.11                     py27_0    defaults
nltk                      3.2                      py27_0    defaults
nose                      1.3.7                    py27_0    defaults
notebook                  4.1.0                    py27_2    defaults
numba                     0.24.0              np110py27_0    defaults
numexpr                   2.5                 np110py27_0    defaults
numpy                     1.10.4                   py27_0    defaults
odo                       0.4.2                    py27_0    defaults
openpyxl                  2.3.2                    py27_0    defaults
openssl                   1.0.2g                    vc9_0  [vc9]  defaults
pandas                    0.17.1              np110py27_0    defaults
pandas-datareader         0.2.1                    py27_0    defaults
partd                     0.3.2                    py27_1    defaults
path.py                   8.1.2                    py27_1    defaults
patsy                     0.4.0               np110py27_0    defaults
pep8                      1.7.0                    py27_0    defaults
pickleshare               0.5                      py27_0    defaults
pillow                    3.1.1                    py27_0    defaults
pip                       8.1.1                    py27_1    defaults
ply                       3.8                      py27_0    defaults
prettytable               0.7.2                     <pip>
psutil                    4.1.0                    py27_0    defaults
py                        1.4.31                   py27_0    defaults
pyasn1                    0.1.9                    py27_0    defaults
pycosat                   0.6.1                    py27_0    defaults
pycparser                 2.14                     py27_0    defaults
pycrypto                  2.6.1                    py27_3    defaults
pyflakes                  1.1.0                    py27_0    defaults
pygments                  2.1.1                    py27_0    defaults
pyopenssl                 0.15.1                   py27_2    defaults
pyparsing                 2.0.3                    py27_0    defaults
pyqt                      4.11.4                   py27_5    defaults
pyreadline                2.1                      py27_0    defaults
pytables                  3.2.2               np110py27_2    defaults
pytest                    2.8.5                    py27_0    defaults
python                    2.7.11                        4    defaults
python-dateutil           2.5.1                    py27_0    defaults
pytz                      2016.2                   py27_0    defaults
pywin32                   220                      py27_1    defaults
pyyaml                    3.11                     py27_3    defaults
pyzmq                     15.2.0                   py27_0    defaults
qt                        4.8.7                     vc9_7  [vc9]  defaults
qtawesome                 0.3.2                    py27_0    defaults
qtconsole                 4.2.0                    py27_1    defaults
qtpy                      1.0                      py27_0    defaults
quandl                    3.0.0                     <pip>
requests                  2.9.1                    py27_0    defaults
requests-file             1.4                      py27_0    defaults
rope                      0.9.4                    py27_1    defaults
rsa                       3.1.2                     <pip>
scikit-image              0.12.3              np110py27_0    defaults
scikit-learn              0.17.1              np110py27_0    defaults
scipy                     0.17.0              np110py27_0    defaults
setuptools                20.3                     py27_0    defaults
simplegeneric             0.8.1                    py27_0    defaults
simplejson                3.8.2                     <pip>
singledispatch            3.4.0.3                  py27_0    defaults
sip                       4.16.9                   py27_2    defaults
six                       1.10.0                   py27_0    defaults
snowballstemmer           1.2.1                    py27_0    defaults
sockjs-tornado            1.0.1                    py27_0    defaults
sortedcontainers          1.4.4                    py27_0    Quantopian
sphinx                    1.3.5                    py27_0    defaults
sphinx-rtd-theme          0.1.9                     <pip>
sphinx_rtd_theme          0.1.9                    py27_0    defaults
spyder                    2.3.8                    py27_1    defaults
sqlalchemy                1.0.12                   py27_0    defaults
sseclient                 0.0.8                     <pip>
ssl_match_hostname        3.4.0.2                  py27_0    defaults
statsmodels               0.6.1               np110py27_0    defaults
sympy                     1.0                      py27_0    defaults
ta-lib                    0.4.9                     <pip>
tables                    3.2.2                     <pip>
tk                        8.5.18                    vc9_0  [vc9]  defaults
toolz                     0.7.4                    py27_0    defaults
tornado                   4.3                      py27_0    defaults
traitlets                 4.2.1                    py27_0    defaults
unicodecsv                0.14.1                   py27_0    defaults
vs2008_runtime            9.00.30729.1                  0    defaults
vs2010_runtime            10.00.40219.1                 0    defaults
werkzeug                  0.11.4                   py27_0    defaults
wheel                     0.29.0                   py27_0    defaults
xlrd                      0.9.4                    py27_0    defaults
xlsxwriter                0.8.4                    py27_0    defaults
xlwings                   0.7.0                    py27_0    defaults
xlwt                      1.0.0                    py27_0    defaults
yahoo-finance             1.2.1                     <pip>
zipline                   1.0.0               np110py27_0    Quantopian
zlib                      1.2.8                     vc9_2  [vc9]  defaults

Description of Issue

I'm running a Jupyer Notebook with a simple test algo with 1 minute data of SPY loaded from a CSV local file. Data is structured as a Panel as:

<class 'pandas.core.panel.Panel'>
Dimensions: 1 (items) x 1000 (major_axis) x 5 (minor_axis)
Items axis: SPY to SPY
Major_axis axis: 2006-11-06 09:31:00+00:00 to 2006-11-08 13:10:00+00:00
Minor_axis axis: open to volume

and has a Date index in UTC timezone. Minor axis has the following items:

Index([u'open', u'high', u'low', u'close', u'volume'], dtype='object')

Running this simple algo as following:

import pytz
import pandas as pd
from datetime import datetime

from zipline.algorithm import TradingAlgorithm

def initialize(context):
    pass

def handle_data(context, data):
    order(symbol('SPY'), 10)

algo_obj = TradingAlgorithm(initialize=initialize, 
                            handle_data=handle_data, data_frequency = 'minute')

perf_manual = algo_obj.run(data)

I got this error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-275-7b68081af855> in <module>()
     15                             handle_data=handle_data, data_frequency = 'minute')
     16 
---> 17 perf_manual = algo_obj.run(data)

C:\Users\andrea.nalon\AppData\Local\Continuum\Anaconda\lib\site-packages\zipline\algorithm.pyc in run(self, data, overwrite_sim_params)
    594                 )
    595                 copy_panel.items = self._write_and_map_id_index_to_sids(
--> 596                     copy_panel.items, copy_panel.major_axis[0],
    597                 )
    598                 self._assets_from_source = (

C:\Users\andrea.nalon\AppData\Local\Continuum\Anaconda\lib\site-packages\zipline\algorithm.pyc in _write_and_map_id_index_to_sids(self, identifiers, as_of_date)
    731 
    732         return self.asset_finder.map_identifier_index_to_sids(
--> 733             identifiers, as_of_date,
    734         )
    735 

C:\Users\andrea.nalon\AppData\Local\Continuum\Anaconda\lib\site-packages\zipline\assets\assets.pyc in map_identifier_index_to_sids(self, index, as_of_date)
    837 
    838         if missing:
--> 839             raise ValueError("Missing assets for identifiers: %s" % missing)
    840 
    841         # Return a list of the sids of the found assets

ValueError: Missing assets for identifiers: ['SPY']

but if I aggregate my data into data2 with daily timeframe as:

OHLC = ['open', 'high', 'low', 'close', 'volume']
OHLC_DICT = {                                                                                                             
'open':'first',                                                                  
'high':'max',
'low':'min',
'close': 'last',
'volume': 'sum'
}
data2 = data.resample('D', how=OHLC_DICT, closed='left', label='left')
data2 = pd.Panel({'SPY': data2})

and I run it with a daily data_frequency

algo_obj = TradingAlgorithm(initialize=initialize, 
                            handle_data=handle_data, data_frequency = 'daily')

perf_manual = algo_obj.run(data2)

this time it works fine without any error.

It seems that Zipline can't deal with 1 minute data. Any idea to solve that issue?

Sincerely, Andrea Nalon

freddiev4 commented 6 years ago

This should be fixed with #1313