search

scrtlabs / catalyst

An Algorithmic Trading Library for Crypto-Assets in Python
http://enigma.co
Apache License 2.0
2.49k stars 725 forks source link

Catalyst should not crash when timouts occur #324

Open sam31415 opened 6 years ago

sam31415 commented 6 years ago

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Now that you know a little about me, let me tell you about the issue I am having:

Description of Issue

There was a timeout while Catalyst was trying to fetch the balances on Poloniex. Poloniex is apparently under maintenance right now. This caused the algorithm to crash. 

[2018-05-06 11:37:02.432304] WARNING: CCXT: unable to fetch balance poloniex: poloniex {"error":"Internal error. Please try again."}        
[2018-05-06 11:37:11.389414] WARNING: CCXT: unable to fetch balance poloniex: poloniex {"error":"Internal error. Please try again."}                           
[2018-05-06 11:37:28.910313] WARNING: CCXT: unable to fetch balance poloniex: poloniex POST https://poloniex.com/tradingApi HTTPSConnectionPool(host='poloniex.
com', port=443): Read timed out. (read timeout=10)
[2018-05-06 11:37:49.242851] WARNING: CCXT: unable to fetch balance poloniex: poloniex POST https://poloniex.com/tradingApi HTTPSConnectionPool(host='poloniex.
com', port=443): Read timed out. (read timeout=10)
[2018-05-06 11:38:17.206662] WARNING: CCXT: unable to fetch balance poloniex: poloniex POST https://poloniex.com/tradingApi HTTPSConnectionPool(host='poloniex.
com', port=443): Read timed out. (read timeout=10)

Traceback (most recent call last):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    six.raise_from(e, None)
  File "<string>", line 2, in raise_from
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 389, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
    raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='poloniex.com', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 356, in fetch
    proxies=self.proxies
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/requests/adapters.py", line 521, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='poloniex.com', port=443): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 628, in get_balances
    balances = self.api.fetch_balance()
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/poloniex.py", line 227, in fetch_balance
    }, params))
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 305, in request
    return self.fetch2(path, api, method, params, headers, body)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 302, in fetch2
    return self.fetch(request['url'], request['method'], request['headers'], request['body'])
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 366, in fetch
    self.raise_error(RequestTimeout, method, url, e)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/ccxt/base/exchange.py", line 287, in raise_error
    raise exception_type(output)
ccxt.base.errors.RequestTimeout: poloniex POST https://poloniex.com/tradingApi HTTPSConnectionPool(host='poloniex.com', port=443): Read timed out. (read timeou
t=10)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "algo.py", line 855, in <module>
    simulate_orders = PAPER_TRADE,
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", line 584, in run_algorithm
    stats_output=stats_output
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/run_algo.py", line 348, in _run
    overwrite_sim_params=False,
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 1044, in run
    data, overwrite_sim_params
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 358, in run
    data, overwrite_sim_params
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/algorithm.py", line 724, in run
    for perf in self.get_generator():
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py", line 224, in transform
    for capital_change_packet in every_bar(dt):
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/gens/tradesimulation.py", line 137, in every_bar
    handle_data(algo, current_data, dt_to_use)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line 216, in handle_data
    dt,
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/utils/events.py", line 235, in handle_data
    self.callback(context, data)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 890, in handle_data
    cleanup=lambda: log.warn('Ordering again.')
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/redo/__init__.py", line 162, in retry
    return action(*args, **kwargs)
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange_algorithm.py", line 713, in synchronize_portfolio
    cash=required_cash,
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/exchange.py", line 702, in sync_positions
    balances = self.get_balances()
  File "/home/user/miniconda3/envs/catalyst/lib/python3.6/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 640, in get_balances
    raise ExchangeRequestError(error=e)
catalyst.exchange.exchange_errors.ExchangeRequestError: Request failed: poloniex POST https://poloniex.com/tradingApi HTTPSConnectionPool(host='poloniex.com',
port=443): Read timed out. (read timeout=10)

I feel that in the case of a timeout, Catalyst should not crash, but rather just continue trying to connect while providing warnings, maybe with a decreasing frequency, until the problem disappears. You don't want a trading algorithm to just crash, leaving positions open.

I had similar issues when fetching price data, which I solved by explicitly catching the exceptions and taking appropriate action in my algorithm. But the problem above lie outside the scope of the algorithm.

Here is how you can reproduce this issue on your machine:

Reproduction Steps

  1. Wait for a timeout... Or maybe cut off your internet connection for a while.

Sincerely, Samuel

lenak25 commented 6 years ago

Hi @sam31415 ,

Thanks for reporting this. The default retries number is 5, which can be reseted in your code, however will not solve the problem as you described it. This is a tricky one, as exchanges' maintenance window or downtime can last for several days.

sam31415 commented 6 years ago

Hi,

Thanks, do you mean that I can control this from the algorithm? Or do I need to change my local version of the Catalyst code?

But what is the reason for not having an infinite number of retries, warning simply the user when the attempt fails? I really think that a trading algorithm should crash only if there is no alternative.

lenak25 commented 6 years ago

Hi @sam31415 ,

In the ExchangeTradingAlgorithmBase (exchange_algorithm.py) the following defaults are defined:

self.attempts = dict(
            get_transactions_attempts=5,
            order_attempts=5,
            synchronize_portfolio_attempts=5,
            get_order_attempts=5,
            get_open_orders_attempts=5,
            cancel_order_attempts=5,
            get_spot_value_attempts=5,
            get_history_window_attempts=5,
            retry_sleeptime=5,
        )

They can be reseted in your initialize function by doing something like this (for example): context.attempts["synchronize_portfolio_attempts"]=100.

I agree with you that a trading algorithm should be resilient. Thanks for bringing this up, we will try to address this.

sam31415 commented 6 years ago

Ah great, I didn't know about context.attempts. Thanks!