search

scrtlabs / catalyst

An Algorithmic Trading Library for Crypto-Assets in Python
http://enigma.co
Apache License 2.0
2.49k stars 725 forks source link

Frequent timeouts #248

Open vonpupp opened 6 years ago

vonpupp commented 6 years ago

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Now that you know a little about me, let me tell you about the issue I am having:

Frequent timeouts

When running in paper mode with minute data frequency, I resample the candles every 30 minutes. Quite often, when I resample the data I get the following error:

[2018-02-23 18:15:00.726740] WARNING: exchange_algorithm: unable save stats locally: 'plus_dm'
[2018-02-23 18:16:00.810781] INFO: exchange_algorithm: portfolio balances, cash: 1111.11, positions: 0.0
[2018-02-23 18:16:00.811479] INFO: d_papert_abba_adam: RESAMPLE 8/240: 2018-02-23 18:16:00+00:00
[2018-02-23 18:16:10.825476] INFO: d_papert_abba_adam: Exception on default_handle_data
Traceback (most recent call last):
  File "/home/av/repos/example/strategies/family/abba.py", line 209, in default_handle_data
    print_log(context, msg)
  File "catalyst/_protocol.pyx", line 120, in catalyst._protocol.check_parameters.__call__.assert_keywords_and_call
    return func(*args, **kwargs)
  File "catalyst/_protocol.pyx", line 646, in catalyst._protocol.BarData.history
    df = self.data_portal.get_history_window(
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/exchange_data_portal.py", line 95, in get_history_window
    ffill))
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/redo/__init__.py", line 162, in retry
    return action(*args, **kwargs)
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/exchange_data_portal.py", line 69, in _get_history_window
    ffill)
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/exchange_data_portal.py", line 218, in get_exchange_history_
window
    False)
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/exchange.py", line 511, in get_history_window
    end_dt=end_dt if not is_current else None,
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/catalyst/exchange/ccxt/ccxt_exchange.py", line 456, in get_candles
    params={}
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/ccxt/binance.py", line 536, in fetch_ohlcv
    response = self.publicGetKlines(self.extend(request, params))
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/ccxt/base/exchange.py", line 310, in request
    return self.fetch2(path, api, method, params, headers, body)
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/ccxt/base/exchange.py", line 307, in fetch2
    return self.fetch(request['url'], request['method'], request['headers'], request['body'])
  File "/home/av/repos/example/.envc-unstable/local/lib/python2.7/site-packages/ccxt/base/exchange.py", line 365, in fetch
    raise RequestTimeout(' '.join([self.id, method, url, 'request timeout']))
RequestTimeout: binance GET https://api.binance.com/api/v1/klines?symbol=NEOUSDT&interval=30m&limit=48 request timeout

You might want to allow the user to setup a timeout on Catalyst to be passed to CCXT. Perhaps even better is to export a CCXT interface to the Catalyst users so it is possible to modify internal properties/values. You may want to have a look at this issue on CCXT as a reference.

Here is how you can reproduce this issue on your machine:

Reproduction Steps

  1. Use paper mode
  2. Do resample
  3. Wait

I am working on a minimal example and will provide a gist link.

What steps have you taken to resolve this already?

None. I think it is an architectural thing that I am not able to solve.

Sincerely, V

fredfortier commented 6 years ago

Yes, we can include a setting which allows passes through to the exchange init options of CCXT. We'll look more specifically into timeout interval options.

For now, it looks like this issue occurs in the your handle_data implementation. This means that you have an opportunity to catch and retry.

We've implemented except...retry handlers in all internal components making requests to an exchange(when synchronizing balances for example). However, we have not done so for ad-hoc operations (like get_history). The rationale is that, since those calls occur in a user-defined function, users might want to control when to retry and when to abandon for the current frame.

Let us know if you think that the retry behavior should be built-in, or if you are simply looking for a way to set the timeout interval (assuming that exchanges accept such parameter).

vonpupp commented 6 years ago

Got it Fred, thank you very much.

I will try to catch that on my code and retry to get the history again and see how it goes, it makes sense. I think a way of setting the timeout would be nice.