Environment

* Operating System: (Windows Version or `$ uname --all`): **Windows-10-10.0.18362-SP0** * Python Version: `$ python --version`: **Python 3.5.6 :: Anaconda, Inc.** : Running on Jupyter Lab. * Python Bitness: `$ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'` **64** * How did you install Zipline: (`pip`, `conda`, or `other (please explain)`) **conda clean install within env python=3.5** * Python packages: `$ pip freeze` or `$ conda list` # packages in environment at C:\Users\Jaepil\Anaconda3\envs\finance35: # # Name Version Build Channel alembic 0.7.7 py35_0 quantopian asn1crypto 0.24.0 py35_0 backcall 0.1.0 py35_0 bcolz 0.12.1 np114py35_0 quantopian blas 1.0 mkl bleach 3.1.0 py_0 blosc 1.16.3 h7bd577a_0 bottleneck 1.2.1 py35h452e1ab_1 bzip2 1.0.8 he774522_0 ca-certificates 2019.11.27 0 certifi 2018.8.24 py35_1 cffi 1.11.5 py35h74b6da3_1 chardet 3.0.4 py35_1 click 7.0 py_0 colorama 0.4.3 py_0 contextlib2 0.6.0.post1 py_0 cryptography 2.3.1 py35h74b6da3_0 cycler 0.10.0 py35hcc71164_0 cyordereddict 0.2.2 py35_0 quantopian cython 0.28.5 py35h6538335_0 decorator 4.4.1 py_0 defusedxml 0.6.0 py_0 empyrical 0.5.0 py35_0 quantopian entrypoints 0.2.3 py35_2 freetype 2.9.1 ha9979f8_1 hdf5 1.10.2 hac2f561_1 icc_rt 2019.0.0 h0cc432a_1 icu 58.2 ha66f8fd_1 idna 2.7 py35_0 inflection 0.3.1 py35_1 intel-openmp 2019.4 245 intervaltree 2.1.0 py35_0 quantopian ipykernel 4.10.0 py35_0 ipython 6.5.0 py35_0 ipython_genutils 0.2.0 py35ha709e79_0 jedi 0.12.1 py35_0 jinja2 2.10.3 py_0 jpeg 9b hb83a4c4_2 json5 0.8.5 py_0 jsonschema 2.6.0 py35h27d56d3_0 jupyter_client 5.3.3 py_0 jupyter_core 4.5.0 py_0 jupyterlab 1.2.4 pyhf63ae98_0 jupyterlab_server 1.0.0 py_0 kiwisolver 1.0.1 py35h6538335_0 libiconv 1.15 h1df5818_7 libpng 1.6.37 h2a8f88b_0 libsodium 1.0.16 h9d3ae62_0 libxml2 2.9.9 h464c3ec_0 libxslt 1.1.33 h579f668_0 logbook 0.12.5 py35_0 quantopian lru-dict 1.1.4 py35_0 quantopian lxml 4.2.5 py35hef2cd61_0 lz4-c 1.8.1.2 h2fa13f4_0 lzo 2.10 h6df0209_2 m2w64-gcc-libgfortran 5.3.0 6 m2w64-gcc-libs 5.3.0 7 m2w64-gcc-libs-core 5.3.0 7 m2w64-gmp 6.1.0 2 m2w64-libwinpthread-git 5.0.0.4634.697f757 2 mako 1.1.0 py_0 markupsafe 1.0 py35hfa6e2cd_1 matplotlib 3.0.0 py35hd159220_0 mistune 0.8.3 py35hfa6e2cd_1 mkl 2018.0.3 1 more-itertools 8.0.2 py_0 msys2-conda-epoch 20160418 1 multipledispatch 0.6.0 py35_0 nbconvert 5.5.0 py_0 nbformat 4.4.0 py35h908c9d9_0 networkx 1.11 py35_1 notebook 5.6.0 py35_0 numexpr 2.6.1 np114py35_0 quantopian numpy 1.14.2 py35h5c71026_0 openssl 1.0.2u he774522_0 pandas 0.22.0 py35h6538335_0 pandas-datareader 0.8.1 py_0 pandoc 2.2.3.2 0 pandocfilters 1.4.2 py35_1 parso 0.5.2 py_0 patsy 0.5.0 py35_0 pickleshare 0.7.4 py35h2f9f535_0 pip 19.2.3 pypi_0 pypi prometheus_client 0.7.1 py_0 prompt_toolkit 1.0.15 py35h89c7cb4_0 pycparser 2.19 py35_0 pygments 2.5.2 py_0 pyopenssl 18.0.0 py35_0 pyparsing 2.4.5 py_0 pyqt 5.9.2 py35h6538335_2 pysocks 1.6.8 py35_0 pytables 3.4.4 py35he6f6034_0 python 3.5.6 he025d50_0 python-dateutil 2.7.3 py35_0 pytz 2019.2 py_0 pywin32 223 py35hfa6e2cd_1 pywinpty 0.5.4 py35_0 pyzmq 17.1.2 py35hfa6e2cd_0 qt 5.9.6 vc14h1e9a669_2 quandl 3.4.2 py35_0 requests 2.19.1 py35_0 requests-file 1.4.3 py35_0 scipy 1.1.0 py35hc28095f_0 send2trash 1.5.0 py35_0 setuptools 40.2.0 py35_0 simplegeneric 0.8.1 py35_2 sip 4.19.8 py35h6538335_0 six 1.11.0 py35_1 snappy 1.1.7 h777316e_3 sortedcontainers 1.4.4 py35_0 quantopian sqlalchemy 1.2.11 py35hfa6e2cd_0 sqlite 3.30.1 he774522_0 statsmodels 0.9.0 py35h452e1ab_0 terminado 0.8.1 py35_1 testpath 0.4.4 py_0 toolz 0.10.0 py_0 tornado 5.1.1 py35hfa6e2cd_0 trading-calendars 1.11.1 py35_0 quantopian traitlets 4.3.2 py35h09b975b_0 urllib3 1.23 py35_0 vc 14.1 h0510ff6_4 vs2015_runtime 14.16.27012 hf0eaf9b_0 wcwidth 0.1.7 py35h6e80d8a_0 webencodings 0.5.1 py35_1 wheel 0.31.1 py35_0 win_inet_pton 1.0.1 py35_1 win_unicode_console 0.5 py35h56988b5_0 wincertstore 0.2 py35hfebbdb8_0 winpty 0.4.3 4 xz 5.2.4 h2fa13f4_4 zeromq 4.2.5 he025d50_1 zipline 1.3.0 np114py35_0 quantopian zlib 1.2.11 h62dcd97_3 zstd 1.3.7 h508b16e_0

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

What did you expect to happen? I expected the custom csv data to be ingested and my data bundle to be registered.
What happened instead? It seems the data bundle has been added to bundle but when I try to ingest it, it says there's no such bundle registered.

Here is how you can reproduce this issue on your machine:

Reproduction Steps

Get AAPL data from yahoo, using pandas_datareader

import pandas as pd
import pandas_datareader.data as web
from zipline.api import order, symbol, record
from zipline.data.bundles import register
from zipline.data.bundles.csvdir import csvdir_equities
import datetime
import matplotlib.pyplot as plt
import os, sys, platform

start = datetime.datetime(2015, 1, 1)
end = datetime.datetime(2020, 1, 10)

data = web.DataReader('AAPL', 'yahoo', start, end)

Change column names to fit zipline's data format (OHLCV + dividend, split)

data = data[['Open', 'High', 'Low', 'Adj Close', 'Volume']] # To OHLCV format. 
data = data.rename(columns={'Open':'open', 'High':'high', 'Low':'low', 'Adj Close':'close', 'Volume':'volume'})
data.index.names = ['date']
data['dividend'] = 0
data['split'] = 1

data.tail()

    open    high    low close   volume  dividend    split
date                            
2020-01-06  293.790009  299.959991  292.750000  299.799988  29596800.0  0   1
2020-01-07  299.839996  300.899994  297.480011  298.390015  27218000.0  0   1
2020-01-08  297.160004  304.440002  297.160004  303.190002  33019800.0  0   1
2020-01-09  307.239990  310.429993  306.200012  309.630005  42527100.0  0   1
2020-01-10  310.600006  312.670013  308.250000  310.329987  35161200.0  0   1

data.head()

    open    high    low close   volume  dividend    split
date                            
2014-12-31  112.820000  113.129997  110.209999  101.419060  41403400.0  0   1
2015-01-02  111.389999  111.440002  107.349998  100.454300  53204600.0  0   1
2015-01-05  108.290001  108.650002  105.410004  97.624336   64285500.0  0   1
2015-01-06  106.540001  107.430000  104.629997  97.633545   65797100.0  0   1
2015-01-07  107.199997  108.199997  106.699997  99.002556   40105900.0  0   1

Export it to csv
```
data.to_csv('AAPL.csv')
```
Follow custom csv ingesting tutorial in the documentation, then you get an error.

# We’ll then want to specify the start and end sessions of our bundle data:
start_session = pd.Timestamp('2014-12-31', tz='utc')
end_session = pd.Timestamp('2020-01-10', tz='utc')

AAPL_path = os.getcwd() + '\AAPL.csv'
print(AAPL_path)
>> 
E:\VSCodeProjects\BOOK-python_algorithm_trading\64-bit-jupyter_notes\AAPL.csv

# And then we can register() our bundle, and pass the location of the directory in which our .csv files exist:
register(
    'custom-csvdir-bundle',
    csvdir_equities(
        ['daily'],
        AAPL_path,
    ),
    calendar_name='NYSE', # US equities
    start_session=start_session,
    end_session=end_session
)

C:\Users\Jaepil\Anaconda3\envs\finance35\lib\site-packages\ipykernel_launcher.py:11: UserWarning: Overwriting bundle with name 'custom-csvdir-bundle' #This is added back by InteractiveShellApp.init_path()

! zipline ingest -b custom-csvdir-bundle 
# Which is equivalent to: $ zipline ingest -b ...

Error: No bundle registered with the name 'custom-csvdir-bundle'

...

What steps have you taken to resolve this already?

I can't understand how my custom bundle could be overwritten (which indicates that it has already existed) and not have been registered at the same time.

I searched existing issues and tried this to change C:\Users\Jaepil\Anaconda3\envs\finance35\Lib\site-packages\zipline\data\bundles\csvdir.py

And so I changed the line from . import core as bundles to from zipline.data.bundles import core as bundles; the extension.py doesn't know about the . the quandl bundle is referencing because the extension code lives in .zipline/.

However, it didn't work. @freddiev4 seems to have managed to make 'No bundle' issue go away but I didn't even get there.

...

Anything else?

On zipline's official documentation, it says:

Once you have your data in the correct format, you can edit your extension.py file in ~/.zipline/extension.py and import the csvdir bundle, along with pandas.

However, it is extremely unclear how I should edit my extension.py. I really could use some help.

...

Sincerely, $ whoami

jaepil-pc\jaepil

eolszewski commented 4 years ago

bump

gpeevans commented 3 years ago

Hi, here are a few comments as I faced a similar issue.

Following the example for loading the custom bundle, firstly from the cli simply run zipline ingest. This should download the quantopian-quandl bundle to the zipline root folder. In windows this was "C:\Users\user.zipline" and an "extension.py" file appeared in this folder. Update the below code as advised by the how-to guide in that file and from the cli run "python extension.py" to ingest the csv data that you prepared.

Note, it is a bit tricky working between Jupyter and the cli as "zipline bundle" from the cli will show registered bundles however "bundle.bundles" when called from Jupyter will not show the same registered even when working in the same conda env. Stepping through the My First Algorithm part helped clear this up a little, accessing the custom csv bundle via cell magic worked a treat.

Cheers

import pandas as pd
from zipline.data.bundles import register
from zipline.data.bundles.csvdir import csvdir_equities
start_session = pd.Timestamp('2012-1-3', tz='utc')
end_session = pd.Timestamp('2014-12-31', tz='utc')
register(
    'custom-csvdir-bundle',
    csvdir_equities(
        ['daily'],
        r'C:\temp\csvData',
    ),
    calendar_name='NYSE', # US equities
    start_session=start_session,
    end_session=end_session

quantopian / zipline

Error: No bundle registered - even though it actually is. #2615

Environment

Description of Issue

Reproduction Steps

What steps have you taken to resolve this already?

Anything else?