scrtlabs / catalyst

An Algorithmic Trading Library for Crypto-Assets in Python

http://enigma.co

Apache License 2.0

2.49k stars 726 forks source link

Can't ingest bitfinex data #380

Open chairy11 opened 6 years ago

chairy11 commented 6 years ago

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Operating System: macOS Sierra 10.12.4
Python Version: Python 3.6.4 :: Anaconda, Inc.
How did you install Catalyst: conda
Python packages: ccxt 1.12.131

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

Expect： get data before backtest by running catalyst ingest-exchange -x bitfinex -i btc_usd

What happened instead? raise error


Trying to ingest exchange bundle bitfinex...
[2018-07-01 12:45:48.908644] WARNING: CCXT: unable to fetch markets bitfinex: bitfinex GET https://api.bitfinex.com/v1/symbols_details HTTPSConnectionPool(host='api.bitfinex.com', port=443): Max retries exceeded with url: /v1/symbols_details (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)'))
Traceback (most recent call last):
File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
(self.host, self.port), self.timeout, **extra_kw)
File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
raise err
File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request self._validate_conn(conn) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn conn.connect() File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connection.py", line 284, in connect conn = self._new_conn() File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connection.py", line 146, in _new_conn (self.host, self.timeout)) urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/requests/adapters.py", line 440, in send timeout=timeout File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.bitfinex.com', port=443): Max retries exceeded with url: /v1/symbols_details (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 356, in fetch proxies=self.proxies File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/requests/sessions.py", line 508, in request resp = self.send(prep, send_kwargs) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/requests/sessions.py", line 618, in send r = adapter.send(request, kwargs) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/requests/adapters.py", line 496, in send raise ConnectTimeout(e, request=request) requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='api.bitfinex.com', port=443): Max retries exceeded with url: /v1/symbols_details (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 108, in init markets_symbols = self.api.load_markets() File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 870, in load_markets markets = self.fetch_markets() File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/bitfinex.py", line 297, in fetch_markets markets = self.publicGetSymbolsDetails() File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 305, in request return self.fetch2(path, api, method, params, headers, body) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 302, in fetch2 return self.fetch(request['url'], request['method'], request['headers'], request['body']) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 366, in fetch self.raise_error(RequestTimeout, method, url, e) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 287, in raise_error raise exception_type(output) ccxt.base.errors.RequestTimeout: bitfinex GET https://api.bitfinex.com/v1/symbols_details HTTPSConnectionPool(host='api.bitfinex.com', port=443): Max retries exceeded with url: /v1/symbols_details (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/chairy11/anaconda3/envs/catalyst/bin/catalyst", line 11, in sys.exit(main()) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/core.py", line 722, in call return self.main(args, kwargs) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(args, *kwargs) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func return f(get_current_context(), args, **kwargs) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/main.py", line 609, in ingest_exchange csv=csv File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_bundle.py", line 811, in ingest self.exchange = get_exchange(self.exchange_name) File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/utils/factory.py", line 43, in get_exchange exchange.init() File "/Users/chairy11/anaconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 125, in init raise ExchangeRequestError(error=e) catalyst.exchange.exchange_errors.ExchangeRequestError: Request failed: bitfinex GET https://api.bitfinex.com/v1/symbols_details HTTPSConnectionPool(host='api.bitfinex.com', port=443): Max retries exceeded with url: /v1/symbols_details (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x114e242e8>, 'Connection to api.bitfinex.com timed out. (connect timeout=10)'))



Here is how you can reproduce this issue on your machine:

## Reproduction Steps

1. `vi ~/.catalyst/data/exchanges/bitfinex/auth.json ` and set the bitfinex API key
2. run `catalyst ingest-exchange -x bitfinex -i btc_usd`
...

## What steps have you taken to resolve this already?
1. I try to open the url `https://api.bitfinex.com/v1/symbols_details` in my brower and get the right data.
![image](https://user-images.githubusercontent.com/1669457/42134573-daf2f280-7d70-11e8-9794-b4c658daff08.png)
2. I try to run `catalyst ingest-exchange -x bittrex -i btc_usdt`, and it seems ok.
![image](https://user-images.githubusercontent.com/1669457/42134582-399e09f0-7d71-11e8-836e-02ef988ca017.png)

Did I do anything wrong?

Sincerely,
chairy11

lenak25 commented 6 years ago

Thanks for opening a new issue. Although the ingested historical data is downloaded from catalyst servers, as part of catalyst initialization, the supported markets are still fetched directly from the exchange (using the CCXT library), which seems to fail in your case. You are following the instructions correctly, from the stack trace, it seems you are experiencing some connectivity issues with Bitfinex.

chairy11 commented 6 years ago

@lenak25 I meet the same iusse on binance with command catalyst ingest-exchange -x binance -i btc_usdt. I have no idea how to solve this problem.
Could I get the history data with another way?

lenak25 commented 6 years ago

Hi @chairy11 , sorry for the delay in the response. Are you still experiencing these issues?