search

scrtlabs / catalyst

An Algorithmic Trading Library for Crypto-Assets in Python
http://enigma.co
Apache License 2.0
2.47k stars 721 forks source link

Standardize OHLCV frequencies to CCXT timeframes #152

Open lacabra opened 6 years ago

lacabra commented 6 years ago

Dear Catalyst Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Now that you know a little about me, let me tell you about the issue I am having:

data.history() works fine in backtesting mode, but breaks in live mode (paper-trading), which renders unusable if one can't query historical data. Take this simple algo as an example:

import pandas as pd

from catalyst import run_algorithm
from catalyst.api import  symbol

def initialize(context):
    context.asset=symbol('btc_usdt')

def handle_data(context, data):
    df = data.history(context.asset,
                      'close',
                      bar_count=10,
                      frequency='5m',
                     )

if __name__ == '__main__':
    LIVE = True
    if LIVE:
        run_algorithm(
                capital_base=1,
                initialize=initialize,
                handle_data=handle_data,
                exchange_name='poloniex',
                algo_namespace='test_algo',
                base_currency='usdt',
                live=True,
                simulate_orders=True,
            )
    else:
        run_algorithm(
                capital_base=1,
                data_frequency='minute',
                initialize=initialize,
                handle_data=handle_data,
                exchange_name='poloniex',
                algo_namespace='test_algo',
                base_currency='usdt',
                live=False,
                start=pd.to_datetime('2017-12-1', utc=True),
                end=pd.to_datetime('2017-12-1', utc=True),
            )

and switch the LIVE flag between True and False

I would get candle stick data for the requested period

I get the following error in live mode:

[2018-01-11 03:53:36.801247] INFO: exchange_algorithm: initialized trading algorithm in live mode
[2018-01-11 03:53:38.291055] INFO: exchange_algorithm: portfolio balances, cash: 1.0, positions: 0.0
Traceback (most recent call last):
  File "error.py", line 33, in <module>
    simulate_orders=True,
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/utils/run_algo.py", line 527, in run_algorithm
    stats_output=stats_output
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/utils/run_algo.py", line 308, in _run
    overwrite_sim_params=False,
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/exchange/exchange_algorithm.py", line 297, in run
    data, overwrite_sim_params
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/algorithm.py", line 724, in run
    for perf in self.get_generator():
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/gens/tradesimulation.py", line 224, in transform
    for capital_change_packet in every_bar(dt):
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/gens/tradesimulation.py", line 137, in every_bar
    handle_data(algo, current_data, dt_to_use)
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/utils/events.py", line 216, in handle_data
    dt,
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/utils/events.py", line 235, in handle_data
    self.callback(context, data)
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/exchange/exchange_algorithm.py", line 738, in handle_data
    self._handle_data(self, data)
  File "error.py", line 15, in handle_data
    frequency='5m',
  File "catalyst/_protocol.pyx", line 120, in catalyst._protocol.check_parameters.__call__.assert_keywords_and_call
  File "catalyst/_protocol.pyx", line 646, in catalyst._protocol.BarData.history
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/exchange/exchange_data_portal.py", line 95, in get_history_window
    ffill))
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/env-c/lib/python2.7/site-packages/redo/__init__.py", line 162, in retry
    return action(*args, **kwargs)
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/exchange/exchange_data_portal.py", line 69, in _get_history_window
    ffill)
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/exchange/exchange_data_portal.py", line 218, in get_exchange_history_window
    False)
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/catalyst-dev/catalyst/exchange/exchange.py", line 525, in get_history_window
    last_traded = asset_series.index[-1]
  File "/Volumes/Data/Users/victoris/Desktop/Enigma/user-install/env-c/lib/python2.7/site-packages/pandas/tseries/base.py", line 251, in __getitem__
    val = getitem(key)
IndexError: index -1 is out of bounds for axis 0 with size 0

What steps have you taken to resolve this already?

It doesn't seem to be exchange-specific, as multiple exchanges throw the same error

Sincerely,

Victor

fredfortier commented 6 years ago

Yes, 5m wouldn't work. Try 5T to respect the offset alias of Pandas. We can always make this a valid alias, but Pandas would consider it 5 months at the moment.

We should at least make the error message much more explicit.

lacabra commented 6 years ago

I confirm that it works with 5T

In essence, this issue is related to #114 and the need to standardize the timeframes between what CCXT and Catalyst support, and making them both fully interoperable, as right now the transition between backtesting and paper trading is not as smooth as it could be.

fredfortier commented 6 years ago

Yes, I agree that accepting CCXT standards might be more intuitive for most users. We'll have to re-map to Pandas standard for resampling which is fine. I'm not sure if we can accept both without conflicts (e.g. 'm' means minutes in one and month in the other).

On Thu, Jan 11, 2018 at 1:54 PM Victor notifications@github.com wrote:

I confirm that it works with 5T

In essence, this issue is related to #114 https://github.com/enigmampc/catalyst/issues/114 and the need to standardize the timeframes between what CCXT and Catalyst support, and making them both fully interoperable, as right now the transition between backtesting and paper trading is not as smooth as it could be.

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/152#issuecomment-357025443, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-QnGxE2ALmWmctQcaf9chFWrCXr4Hks5tJljhgaJpZM4RaSsj .

kroitor commented 6 years ago

The reason behind those particular literals in ccxt is simple: we deduced them from the most common set that appears in candlestick charts mostly everywhere, this makes timeframe labels recognizeable for users who are familiar with exchanges' web interfaces. Hope it makes sense.

fredfortier commented 6 years ago

Yes, it makes sense. Can’t argue with the fact that ‘m’ is more intuitive than ‘T’. :-) On Fri, Jan 12, 2018 at 8:11 AM Igor Kroitor notifications@github.com wrote:

The reason behind those particular literals in ccxt is simple: we deduced them from the most common set that appears in candlestick charts mostly everywhere, this makes timeframe labels recognizeable for users who are familiar with exchanges' web interfaces. Hope it makes sense.

— You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/152#issuecomment-357280442, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-QhX_oWd40L6_nQAqGySm-s59gV8vks5tJ4PMgaJpZM4RaSsj .