Environment

Operating System: Linux s 4.10.0-42-generic #46-Ubuntu SMP Mon Dec 4 14:38:01 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Python Version: python3.5.3
Python Bitness: 64
How did you install Catalyst: pip3
Python packages: aiodns==1.1.1 aiohttp==2.3.9 alabaster==0.7.10 alembic==0.9.7 aniso8601==1.3.0 apparmor==2.11.0 apparmor.click==0.3.18 appdirs==1.4.3 apturl==0.5.2 asn1crypto==0.24.0 astroid==1.5.3 async-timeout==2.0.0 Babel==2.4.0 bcolz==0.12.1 beautifulsoup4==4.6.0 beautysh==3.7 bitcoin==1.1.42 blinker==1.3 boto3==1.5.20 botocore==1.8.34 bottle==0.12.13 Bottleneck==1.2.1 Brlapi==0.6.5 bs4==0.0.1 cchardet==2.1.1 ccxt==1.10.840 certifi==2018.1.18 cffi==1.11.2 chardet==3.0.4 click==6.7 coincurve==6.0.0 command-not-found==0.3 contextlib2==0.5.5 CProfileV==1.0.7 crypto==1.4.1 cryptography==2.1.4 cycler==0.10.0 cyordereddict==1.0.0 Cython==0.27.3 dateparser==0.6.0 decorator==4.2.1 defer==1.0.6 devscripts===2.16.8ubuntu2 discord==0.0.2 discord.py==0.16.12 distro-info===0.14build1 docutils==0.14 dryscrape==1.0.1 EasyProcess==0.2.3 empyrical==0.2.1 enigma-catalyst==0.4.7 enum34==1.1.6 eth-testrpc==1.2.0 ethereum==2.0.4 ethereum-abi-utils==0.4.0 ethereum-serpent==2.0.7 ethereum-utils==0.2.2 feedparser==5.1.3 Flask==0.12.2 Flask-RESTful==0.3.6 future==0.16.0 gevent==1.2.1 greenlet==0.4.12 html5lib==0.999999999 httplib2==0.9.2 humanfriendly==4.7 idna==2.6 imagesize==0.7.1 intervaltree==2.1.0 ipaddress==1.0.18 ipython-genutils==0.2.0 isort==4.2.15 itsdangerous==0.24 Jinja2==2.9.6 jmespath==0.9.3 json-rpc==1.10.3 jsonschema==2.6.0 jupyter-core==4.4.0 jusText==2.2.0 keyring==10.5.1 keyrings.alt==2.2 kik==1.5.0 langdetect==1.0.7 language-selector==0.1 launchpadlib==1.10.4 lazr.restfulclient==0.13.4 lazr.uri==1.0.3 lazy-object-proxy==1.3.1 LibAppArmor==2.11.0 Logbook==1.1.0 louis==3.0.0 lru-dict==1.1.6 lxml==4.1.1 Mako==1.0.7 MarkupSafe==1.0 matplotlib==2.1.1 mccabe==0.6.1 minidb==2.0.2 mpl-finance==0.10.0 multidict==4.0.0 multipledispatch==0.4.9 Naked==0.1.31 nbformat==4.4.0 networkx==2.1 nltk==3.2.5 numexpr==2.6.4 numpy==1.14.0 oauth==1.0.1 oauthlib==2.0.6 pandas==0.19.2 pandas-datareader==0.5.0 patsy==0.5.0 pbkdf2==1.3 pbr==1.10.0 PeakUtils==1.1.0 pep8==1.7.1 pexpect==4.2.1 Pillow==4.0.0 plotly==2.2.3 pluggy==0.5.2 psutil==5.0.1 ptyprocess==0.5.1 pushbullet.py==0.11.0 py==1.4.34 py-ecc==1.1.3 pyasn1==0.1.9 pycares==2.3.0 pycodestyle==2.3.1 pycparser==2.18 pycrypto==2.6.1 pycryptodome==3.4.6 pycups==1.9.73 pydocstyle==2.1.1 pydub==0.20.0 pyethash==0.1.27 pyflakes==1.6.0 Pygments==2.2.0 pygobject==3.22.0 PyJWT==1.5.3 pylama==7.4.3 pylama-pylint==3.0.1 pylint==1.7.4 pylru==1.0.9 pylxd==0.18.0 pyOpenSSL==17.3.0 pyparsing==2.2.0 PyQt5==5.9.2 pysha3==1.0.2 PySocks==1.6.8 pytest==3.2.3 python-apt==1.4.0b2 python-dateutil==2.6.1 python-debian==0.1.30 python-editor==1.0.3 python-magic==0.4.13 python-telegram-bot==9.0.0 python-twitter==3.3 pytrends==4.3.0 pytz==2017.3 PyVirtualDisplay==0.2.1 pyxdg==0.25 PyYAML==3.12 raven==6.4.0 raven-aiohttp==0.6.0 redo==1.6 regex==2018.1.10 reportlab==3.3.0 repoze.lru==0.7 requests==2.18.4 requests-file==1.4.3 requests-ftp==0.3.1 requests-oauthlib==0.8.0 requests-unixsocket==0.1.5 rlp==0.6.0 ruamel.yaml==0.15.35 s3transfer==0.1.12 scikit-learn==0.19.1 scipy==1.0.0 scrypt==0.8.0 seaborn==0.8.1 SecretStorage==2.3.1 selenium==3.4.3 sessioninstaller==0.0.0 shellescape==3.4.1 simplejson==3.10.0 sip==4.19.6 six==1.11.0 sklearn==0.0 sky==0.0.198 slackclient==1.1.0 snowballstemmer==1.2.1 socketclusterclient==1.2.2 sortedcontainers==1.5.9 Sphinx==1.6.2 sphinxcontrib-websupport==1.0.1 SQLAlchemy==1.2.2 ssh-import-id==5.6 statsmodels==0.8.0 system-service==0.3 systemd-python==233 TA-Lib==0.4.10 tables==3.4.2 telegram==0.0.1 textblob==0.15.1 tinydb==3.3.1 tldextract==2.2.0 toolz==0.9.0 tornado==4.5.2 tox==2.9.1 traitlets==4.3.2 twilio==6.6.3 tzlocal==1.5.1 ubuntu-drivers-common==0.0.0 ufw==0.35 unattended-upgrades==0.1 unicodecsv==0.14.1 unity-scope-calculator==0.1 unity-scope-chromiumbookmarks==0.1 unity-scope-colourlovers==0.1 unity-scope-devhelp==0.1 unity-scope-firefoxbookmarks==0.1 unity-scope-gdrive==0.7 unity-scope-manpages==0.1 unity-scope-openclipart==0.1 unity-scope-texdoc==0.1 unity-scope-tomboy==0.1 unity-scope-virtualbox==0.1 unity-scope-yelp==0.1 unity-scope-zotero==0.1 unity-tweak-tool==0.0.7 urllib3==1.22 urlwatch==2.7 usb-creator==0.3.3 vaderSentiment==2.5 virtualenv==15.1.0 wadllib==1.3.2 web3==3.8.0 webencodings==0.5 websocket-client==0.45.0 websockets==3.4 Werkzeug==0.12.2 wrapt==1.10.11 ws4py==0.3.4 xkit==0.0.0 yarl==1.1.0 zope.interface==4.3.2

Now that you know a little about me, let me tell you about the issue I am having:

Exchanging data from binance/kucoin and probably all the other exchanges that are not supported for trading:

Doesn't show a progress bar when downloading. It only shows that it is finished, no way to know if it has hanged.
It doesn't create the symbols.json in the .catalyst/data/exchanges directory so catalyst run just fails silently. I found this is the reason it fails by looking hopelessly through strace. I am open to better suggestions for debugging. xD
If i copy it from poloniex it calls initialize but then fails cause the symbols are different between the exchanges. This is to be expected but it shows the symbols.json absence is the problem.

Here is how you can reproduce this issue on your machine:

catalyst ingest-exchange -x binance -f daily

Sincerely, $ whoami

fredfortier commented 6 years ago

Yes, only poloniex, bitfinex and bittrex (daily only) are supported. This is going to change very soon.

sekamaneka commented 6 years ago

Ok thanks for the clarification. Since the ingesting was actually downloading stuff i thought it must have been added quietly since the ccxt integration.

briannewtonpsyd commented 6 years ago

@fredfortier Hey Fred, I know you guys were going full steam at that first Engima release, and bugfixing on Catalyst, but is there an ETA on Binance ingesting? I've been trading on Poloniex this whole time, but running into random issues, and only trading there because it's available for ingesting, would much prefer Binance. Thanks!

fredfortier commented 6 years ago

Yes, we’re later than expected on this milestone because of the reasons which you mentioned. But we now have a much bigger team and we are catching up. We’re just putting the final touches on the branch which includes Binance data and testing. We’ll release very soon. On Thu, Feb 15, 2018 at 7:09 AM brinew27 notifications@github.com wrote:

@fredfortier https://github.com/fredfortier Hey Fred, I know you guys were going full steam at that first Engima release, and bugfixing on Catalyst, but is there an ETA on Binance ingesting? I've been trading on Poloniex this whole time, but running into random issues, and only trading there because it's available for ingesting, would much prefer Binance. Thanks!

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/188#issuecomment-365955401, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-QnFuLlMVm3n3p9_kX1pYgnA9-bGcks5tVEiQgaJpZM4Rs0nw .

lenak25 commented 6 years ago

Binance historical data was added at release 0.5.15

scrtlabs / catalyst

Ingesting of data from ccxt exchanges has some errors. #188

Environment