quantopian / zipline

Zipline, a Pythonic Algorithmic Trading Library
https://www.zipline.io
Apache License 2.0
17.7k stars 4.73k forks source link

Zipline My First Algorithm not working -- all interesting fields are zero in the pickle file #2721

Open trevor-richardson opened 4 years ago

trevor-richardson commented 4 years ago

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

* Operating System: Ubuntu 18.04 * Python Version: I've tried 3.4 and 2.7 * Python Bitness: both are 64 bit * How did you install Zipline: tried pip and conda * Python packages: # Name Version Build Channel _libgcc_mutex 0.1 main alembic 0.7.7 py35_0 Quantopian asn1crypto 0.24.0 py35_0 backcall 0.1.0 py_0 conda-forge bcolz 0.12.1 np114py35_0 Quantopian beautifulsoup4 4.9.1 pypi_0 pypi blas 1.0 mkl bleach 3.1.5 pyh9f0ad1d_0 conda-forge blosc 1.16.3 hd408876_0 bottleneck 1.2.1 py35h035aef0_1 bzip2 1.0.8 h7b6447c_0 ca-certificates 2020.4.5.1 hecc5488_0 conda-forge certifi 2018.8.24 py35_1001 conda-forge cffi 1.11.5 py35he75722e_1 chardet 3.0.4 py35_1 click 7.1.2 py_0 contextlib2 0.6.0.post1 py_0 cryptography 2.3.1 py35hc365091_0 cycler 0.10.0 py35hc4d5149_0 cyordereddict 0.2.2 py35_0 Quantopian cython 0.28.5 py35hf484d3e_0 dbus 1.13.14 hb2f20db_0 decorator 4.4.2 py_0 defusedxml 0.6.0 py_0 conda-forge empyrical 0.5.0 py35_0 Quantopian entrypoints 0.2.3 py35_2 conda-forge expat 2.2.6 he6710b0_0 fontconfig 2.13.0 h9420a91_0 freetype 2.9.1 h8a8886c_1 glib 2.63.1 h5a9c865_0 gst-plugins-base 1.14.0 hbbd80ab_1 gstreamer 1.14.0 hb453b48_1 hdf5 1.10.2 hba1933b_1 icu 58.2 he6710b0_3 idna 2.7 py35_0 inflection 0.4.0 pypi_0 pypi intel-openmp 2019.4 243 intervaltree 2.1.0 py35_0 Quantopian ipykernel 5.1.0 pyh24bf2e0_0 conda-forge ipython 7.0.1 py35h24bf2e0_0 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jedi 0.12.1 py35_0 conda-forge jinja2 2.11.2 pyh9f0ad1d_0 conda-forge jpeg 9b h024ee3a_2 json5 0.9.0 py_0 conda-forge jsonschema 2.6.0 py35_2 conda-forge jupyter_client 5.3.3 py_0 conda-forge jupyter_core 4.5.0 py_0 conda-forge jupyterlab 2.1.0 py_0 conda-forge jupyterlab_server 1.0.0 py_0 kiwisolver 1.0.1 py35hf484d3e_0 libedit 3.1.20181209 hc058e9b_0 libffi 3.2.1 hd88cf55_4 libgcc-ng 9.1.0 hdf63c60_0 libgfortran-ng 7.3.0 hdf63c60_0 libpng 1.6.37 hbc83047_0 libsodium 1.0.18 h516909a_0 conda-forge libstdcxx-ng 9.1.0 hdf63c60_0 libuuid 1.0.3 h1bed415_2 libxcb 1.13 h1bed415_1 libxml2 2.9.9 hea5a465_1 libxslt 1.1.33 h7d1a2b0_0 logbook 0.12.5 py35_0 Quantopian lru-dict 1.1.4 py35_0 Quantopian lxml 4.2.5 py35hefd8a0e_0 lz4-c 1.8.1.2 h14c3975_0 lzo 2.10 h7b6447c_2 mako 1.1.2 py_0 markupsafe 1.0 py35h14c3975_1 matplotlib 3.0.0 py35h5429711_0 mistune 0.8.3 py35h470a237_2 conda-forge mkl 2018.0.3 1 mkl_fft 1.0.6 py35h7dd41cf_0 mkl_random 1.0.1 py35h4414c95_1 more-itertools 8.3.0 pypi_0 pypi multipledispatch 0.6.0 py35_0 nbconvert 5.6.0 py_0 conda-forge nbformat 5.0.6 py_0 conda-forge ncurses 6.2 he6710b0_1 networkx 1.11 py35_1 notebook 5.7.0 py35_0 conda-forge numexpr 2.6.1 np114py35_0 Quantopian numpy 1.14.6 py35h3b04361_4 numpy-base 1.14.6 py35h81de0dd_4 openssl 1.0.2u h516909a_0 conda-forge packaging 20.4 pyh9f0ad1d_0 conda-forge pandas 0.22.0 py35hf484d3e_0 pandas-datareader 0.8.1 py_0 pandoc 2.9.2.1 0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.7.0 pyh9f0ad1d_0 conda-forge patsy 0.5.0 py35_0 pcre 8.43 he6710b0_0 pexpect 4.6.0 py35_0 conda-forge pickleshare 0.7.5 py35_0 conda-forge pip 10.0.1 py35_0 prometheus_client 0.8.0 pyh9f0ad1d_0 conda-forge prompt_toolkit 2.0.10 py_0 conda-forge ptyprocess 0.6.0 py_1001 conda-forge pycparser 2.20 py_0 pygments 2.6.1 py_0 conda-forge pyopenssl 18.0.0 py35_0 pyparsing 2.4.7 py_0 pyqt 5.9.2 py35h05f1152_2 pysocks 1.6.8 py35_0 pytables 3.4.4 py35ha205bf6_0 python 3.5.6 hc3d631a_0 python-dateutil 2.8.1 py_0 pytz 2020.1 py_0 pyzmq 17.1.2 py35hae99301_0 conda-forge qt 5.9.6 h8703b6f_2 quandl 3.5.0 pypi_0 pypi readline 7.0 h7b6447c_5 requests 2.20.1 py35_0 Quantopian requests-file 1.4.3 py35_0 scipy 1.1.0 py35hd20e5f9_0 send2trash 1.5.0 py_0 conda-forge setuptools 40.2.0 py35_0 simplegeneric 0.8.1 py_1 conda-forge sip 4.19.8 py35hf484d3e_0 six 1.11.0 py35_1 snappy 1.1.7 hbae5bb6_3 sortedcontainers 2.1.0 py35_0 Quantopian soupsieve 2.0.1 pypi_0 pypi sqlalchemy 1.2.11 py35h7b6447c_0 sqlite 3.31.1 h62c20be_1 statsmodels 0.9.0 py35h3010b51_0 tbb 2020.0 hfd86e86_0 tbb4py 2018.0.5 py35h6bb024c_0 terminado 0.8.1 py35_1 conda-forge testpath 0.4.4 py_0 conda-forge tk 8.6.8 hbc83047_0 toolz 0.10.0 py_0 tornado 5.1.1 py35h7b6447c_0 trading-calendars 1.11.2 py35_0 Quantopian traitlets 4.3.2 py35_0 conda-forge urllib3 1.23 py35_0 watermark 2.0.2 pypi_0 pypi wcwidth 0.2.3 pyh9f0ad1d_0 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.31.1 py35_0 xz 5.2.5 h7b6447c_0 yahoofinancials 1.5 pypi_0 pypi zeromq 4.2.5 hfc679d8_4 conda-forge zipline 1.3.0 np114py35_0 Quantopian zlib 1.2.11 h7b6447c_3 zstd 1.3.7 h0b5b093_0

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

The intro tutorial run on a custom bundle doesn't work. I had to make a custom bundle because quandl won't accept my emails (it says my work email is not a work email). I followed this tutorial to set up my custom bundle -- https://towardsdatascience.com/introduction-to-backtesting-trading-strategies-7afae611a35e

Here are the first couple of lines from csv file cat to terminal. /AlgorithmicTrading$ cat aapl_daily/AAPL.csv date,open,high,low,close,volume,dividend,split 2017-01-03,115.80000305175781,116.33000183105469,114.76000213623047,116.1500015258789,28781900,0,0 2017-01-04,115.8499984741211,116.51000213623047,115.75,116.0199966430664,21118100,0,0

I know have the following bundles: apple-prices-2017-2019 2020-06-05 12:49:38.522678 apple-prices-2017-2019 2020-06-05 11:47:48.113717 apple-prices-2017-2019 2020-06-05 11:37:22.842459 apple-prices-2017-2019 2020-06-05 11:08:31.692111 ayx-prices-2019-2020 2020-06-04 13:17:20.079053 ayx-prices-2019-2020 2020-06-04 13:16:18.612500 ayx-prices-2019-2020 2020-06-04 12:45:41.304859 csvdir quandl quantopian-quandl

I run the following command on your buyapple example

zipline run -f /anaconda3/pkgs/zipline-1.3.0-np114py35_0/lib/python3.5/site-packages/zipline/examples/buyapple.py --bundle apple-prices-2017-2019 --start 2017-1-1 --end 2018-1-1 -o backtest.pickle

Almost all of the values in the resulting pickle are zero. And the profit is a flat line buyapple

Here is how you can reproduce this issue on your machine:

Reproduction Steps

Made a new conda environment. Installed zipline registered new bundle run zipline run command visualize that nothing happened except for tracking stock price (attached image shows this) ...

What steps have you taken to resolve this already?

I've tried python 3.5 and 2.7. I've tried jupyter notebook. I've tried following this peron's exact blog https://towardsdatascience.com/introduction-to-backtesting-trading-strategies-7afae611a35e

I cannot seem to get the backtest to work using any intro strategies.

Thank you!

...

Sincerely, Trevor Richardson buyapple

cemal95 commented 4 years ago

You can use the csvdir.py and extensions.py files to register your bundle if you have personal data and your algo.

trevor-richardson commented 4 years ago

@cemal95 my bundle is already registered. I used the extensions.py file to register it. Is there anyway to check if it is registered correctly besides what I show above which is the output from zipline bundles?

cemal95 commented 4 years ago

currently I dont know how to check. Did you make sure that your data is in the right file, sometimes they forget to provide the right path? It says in the blog to use this command to check if it is regestered: !zipline bundles

Maybe show your code?

trevor-richardson commented 4 years ago

@cemal95 I didn't write any code. I am using the code inside /zipline/examples/buyapple.py (which is provided by zipline). The repo is inside my anaconda environment folder. I ran the following command which should execute that strategy on my bundle. Here is the command:

zipline run -f /anaconda3/pkgs/zipline-1.3.0-np114py35_0/lib/python3.5/site-packages/zipline/examples/buyapple.py --bundle apple-prices-2017-2019 --start 2017-1-1 --end 2018-1-1 -o ./backtest.pickle

cemal95 commented 4 years ago

Hey sorry for the late reply, I ran the example. I used the bundle='quandl'. And added the run_algorithm() function and it worked. Did you check the data maybe, did you download it?

Schermafbeelding 2020-06-06 om 00 00 20

Also, the example only has specific dates on it, the algo runs in the year 2014. If you look at the last method. So maybe thats why, you provided the data maybe, but the time is wrong?

trevor-richardson commented 4 years ago

The first three lines of the csv file I'm using as a custom bundle is printed out above. That custom bundle has data from 2017-2019. This has been confirmed. I really appreciate your help but I don't think I'm conveying the problem to you properly....we can just close this (unanswered). This isn't progressing for me. I can write the python script myself to backtest (simple loop over the data accounting for dates and such). Writing backtest myself (super simple) will be faster than debugging I think. Thanks!

trevor-richardson commented 4 years ago

@cemal95 I really like the code you pasted! Thanks for helping out on this journey!

cemal95 commented 4 years ago

My pleasure man, also on my THESIS repo, I have a short intro on how to use it on jupyter notebook. Also gonna upload my thesis algos.

trevor-richardson commented 4 years ago

I still haven't figured out how to use custom bundles but I thought you should know your sample code helped me get the initial algorithm running with quandl data! Call this effectively closed. Two weird things. I had to edit benchmarks.py in the /zipline/data/ filepath so that json errors weren't occuring. they talk about this issue on other issues pages sorry I can't remember which one exactly and I had to add the quandl api key to export in .bashrc. It has been a journey to say the least. Not like downloading pytorch or matplotlib which just work.... I will definitely check out your thesis repo. Thanks again!

cemal95 commented 4 years ago

Alrighty, cool man. Wish you all the best bro. Please let me know if something can be done to improve on something :)

belortavirin commented 3 years ago

Hello, guys Im doing my first code, unfortunately Im having a lot problems with my code. I already bundle my data in custom-bundle

#%%
import pandas as pd
import zipline
import pytz
import numpy as np
from analysis import create_benchmark, analyze
from zipline.api import order_percent, symbols, order, record, order_target_percent, symbol, schedule_function, date_rules, time_rules
from zipline.finance import commission
from datetime import datetime
from matplotlib import pyplot as plt, ticker, rc
import trading_calendars as tc
#%%
def initialize(context):

    context.set_commission(commission.PerShare(cost=0.0, min_trade_cost=0))
    context.assets= symbols('HGRE11','KNRI11','BPFF11','XPCM11','MXRF11')
    context.n_assets=len(context.assets)
    context.has_position= False

def handle_data(context, data):

    if not context.has_position:
        for asset in context.assets:
            if data.can_trade(asset):
                order_percent(asset, 1/context.n_assets)
        context.has_position= True
result = zipline.run_algorithm(
    start=start, # Set start
    end=end,  # Set end
    initialize=initialize, # Define startup function
    capital_base=100000, # Set initial capital
    data_frequency = 'daily',  # Set data frequency
    bundle='custom-bundle', # Select bundle
    trading_calendar= tc.get_calendar('BVMF'),
    benchmark_returns=bench_series) 
 fig, ax =plt.subplots(1, 1, sharex=True, figsize=[16,9])
 #Portafolio value
 result.portfolio_value.plot(ax=ax[0])
 ax[0].set_ylabel('Portafolio value in$')
 plt.legend()
 plt.show()

image

However, I got zero values in the result file. Im using jupyter to run Zipline. Any clue about this issue?

image