search

scrtlabs / catalyst

An Algorithmic Trading Library for Crypto-Assets in Python
http://enigma.co
Apache License 2.0
2.49k stars 725 forks source link

[0.3] Ingesting exchange bundle throws IndexError #42

Closed reinka closed 6 years ago

reinka commented 6 years ago

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

alembic 0.9.6 backports.functools-lru-cache 1.4 bcolz 0.12.1 Bottleneck 1.2.1 certifi 2016.2.28 py27_0
certifi 2017.7.27.1 chardet 3.0.4 click 6.7 contextlib2 0.5.5 cycler 0.10.0 cyordereddict 1.0.0 Cython 0.27.1 decorator 4.1.2 empyrical 0.2.1 enigma-catalyst 0.3 idna 2.6 intervaltree 2.1.0 Logbook 1.1.0 lru-dict 1.1.6 Mako 1.0.7 MarkupSafe 1.0 matplotlib 2.1.0 mkl 2017.0.3 0
multipledispatch 0.4.9 networkx 2.0 numexpr 2.6.4 numpy 1.13.1 py27_0
openssl 1.0.2l 0
pandas 0.19.2 pandas-datareader 0.5.0 patsy 0.4.1 pip 9.0.1 py27_1
pudb 2017.1.4 Pygments 2.2.0 pyparsing 2.2.0 python 2.7.13 0
python-dateutil 2.6.1 python-editor 1.0.3 pytz 2017.2 readline 6.2 2
requests 2.18.4 requests-file 1.4.2 requests-ftp 0.3.1 scipy 0.19.1 np113py27_0
setuptools 36.4.0 py27_1
six 1.11.0 sortedcontainers 1.5.7 SQLAlchemy 1.1.14 sqlite 3.13.0 0
statsmodels 0.8.0 subprocess32 3.2.7 TA-Lib 0.4.10 tables 3.4.2 tk 8.5.18 0
toolz 0.8.2 urllib3 1.22 urwid 1.3.1 wheel 0.29.0 py27_0
zlib 1.2.11 0

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

Here is how you can reproduce this issue on your machine:

Reproduction Steps

  1. catalyst ingest-exchange -x poloniex -f minute -i xrp_usdt ...

What steps have you taken to resolve this already?

$ ls -l /Users/administrator/.catalyst/data/exchanges/poloniex/minute_bundle/00/35/003541.bcolz/close/data/

-rw-r--r-- 1 apoehlmann staff 4105 Oct 20 23:49 0.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 1.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 10.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 11.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 12.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 13.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 14.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 15.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 16.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 17.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 18.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 19.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 2.blp -rw-r--r-- 1 apoehlmann staff 3144 Oct 20 23:49 20.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 3.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 4.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 5.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 6.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 7.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 8.blp -rw-r--r-- 1 apoehlmann staff 4064 Oct 20 23:49 __9.blp

I then moved the data folder since it's already from the previous version but this didn't solve the problem, however the logs are now slightly different, the Exception ValueError: ValueError('chunkfile [...]) part (beginning of the previous logs) is gone:

$ catalyst ingest-exchange -x poloniex -f minute -i xrp_usdt Ingesting exchange bundle poloniex... [= ] Fetching poloniex minute candles: : 3% Traceback (most recent call last): File "/anaconda/envs/catalyst/bin/catalyst", line 11, in sys.exit(main()) File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 722, in call return self.main(args, kwargs) File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/anaconda/envs/catalyst/lib/python2.7/site-packages/click/core.py", line 535, in invoke return callback(args, **kwargs) File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/main.py", line 508, in ingest_exchange show_progress=show_progress File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py", line 453, in ingest show_progress) File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py", line 431, in ingest_assets empty_rows_behavior='strip' File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bundle.py", line 232, in ingest_ctable end_dt=end_dt File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bcolz.py", line 51, in load_raw_arrays return self._load_raw_arrays(fields, start_dt, end_dt, sids) File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/exchange/exchange_bcolz.py", line 54, in _load_raw_arrays start_idx = self._find_position_of_minute(start_dt) File "/anaconda/envs/catalyst/lib/python2.7/site-packages/catalyst/data/minute_bars.py", line 1209, in _find_position_of_minute False, File "catalyst/data/_minute_bar_internal.pyx", line 81, in catalyst.data._minute_bar_internal.find_position_of_minute IndexError: Out of bounds on buffer access (axis 0)

fredfortier commented 6 years ago

I was able to reproduce this issue. We are looking into it. In the meanwhile, using a range worked for me: catalyst ingest-exchange -x poloniex -f minute -i xrp_usdt -s 2016-01-01

I suspect that the problem lies with the 2015 bundles which is what we are investigating.

fredfortier commented 6 years ago

I can push the start date even earlier.

This works: catalyst ingest-exchange -x poloniex -f minute -i xrp_usdt -s 2015-4-1

But this fails: catalyst ingest-exchange -x poloniex -f minute -i xrp_usdt -s 2015-1-1

We will get back to you shortly with a complete solution but I hope that this helps in the meanwhile.

avn3r commented 6 years ago

Similar error on my end:

Works: catalyst ingest-exchange -x poloniex -f minute -s 2017-1-1 --show-progress -i xrp_usdt Error: catalyst ingest-exchange -x poloniex -f minute -s 2017-1-1 --show-progress

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://s3.amazonaws.com/enigmaco/catalyst-bundles/exchange-poloniex/poloniex-minute-gno_btc-2017-01.tar.gz

If I specify a pair it works but it doesn't work without specifying a pair.

lacabra commented 6 years ago

This issue has been resolved with commit 3d88d6a2c707adfb4e3791773e1ace942f62f8aa in 0.3.3 release. Please upgrade to the latest version: pip install enigma-catalyst --upgrade. Thank you