search

scrtlabs / catalyst

An Algorithmic Trading Library for Crypto-Assets in Python
http://enigma.co
Apache License 2.0
2.49k stars 725 forks source link

CCXT Exchanges: Binance, KuCoin, Cryptopia #182

Open avn3r opened 6 years ago

avn3r commented 6 years ago

Dear Catalyst Team,

Environment

Description of Issue:

  1. Current Volume is displaying the volume in the market currency instead of the base currency. For example, eth_btc reports current_volume= data.current(coin, 'volume') in eth volume instead of BTC volume. I believe same error occurs in data.history. The need for base currency value is in order to properly scale and compare values.

    • What did you expect to happen? current daily/24h volume in BTC
    • What happened instead? got current daily/24h volume in ETH (market currency)

  2. CCXT Kucoin: Many request error when getting the history of the pairs. Error: ccxt.base.errors.RequestTimeout: kucoin GET https://api.kucoin.com/v1/open/chart/history?from=1516062779&symbol=CAT-BTC&type=30&resolution=30&to=1516667579 request timeout

  3. CCXT Cryptopia: Cannot get all BTC pairs. Mainly because some pair symbols are all numbers and it it converting the symbol into an int instead of a string. Error: AttributeError: 'int' object has no attribute 'lower' catalyst/exchange/ccxt/ccxt_exchange.py", line 534, Line 534 creates the error.

    • What did you expect to happen?
    • What happened instead?

Reproduction Steps

  1. Run code using: python <filename.py> -x <exchange_name> 
    
from datetime import timedelta

import numpy as np import pandas as pd import argparse

from catalyst import run_algorithm from catalyst.exchange.utils.exchange_utils import get_exchange_symbols from catalyst.api import (symbols, )

def initialize(context): context.i = -1 # minute counter context.exchange = context.exchanges.values()[0].name.lower() context.base_currency = context.exchanges.values()[0].base_currency.lower()

print(context.coins)

def handle_data(context, data): context.i += 1 lookback_days = 7 # 7 days

# current date & time in each iteration formatted into a string
now = pd.to_datetime(data.current_dt, utc=True)
date, time = now.strftime('%Y-%m-%d %H:%M:%S').split(' ')
lookback_date = now - timedelta(days=lookback_days)

one_day_in_minutes = 1440  # 60 * 24 assumes data_frequency='minute'
# update universe everyday at midnight
if not context.i % one_day_in_minutes:
    context.universe = universe(context, lookback_date, now)

# get data every 30 minutes
minutes = 30
# get lookback_days of history data: that is 'lookback' number of bins
lookback = one_day_in_minutes / minutes * lookback_days
if not context.i % minutes and context.universe:
    # we iterate for every pair in the current universe
    for coin in context.coins:
        pair = str(coin.symbol)

        # Get 30 minute interval OHLCV data. This is the standard data
        # required for candlestick or indicators/signals. Return Pandas
        # DataFrames. 30T means 30-minute re-sampling of one minute data.
        # Adjust it to your desired time interval as needed.
        current_volume = data.current(coin, 'volume')

        opened = fill(data.history(coin, 'open',
                              bar_count=lookback, frequency='30T')).values
        high = fill(data.history(coin, 'high',
                              bar_count=lookback, frequency='30T')).values
        low = fill(data.history(coin, 'low',
                              bar_count=lookback, frequency='30T')).values
        close = fill(data.history(coin, 'price',
                              bar_count=lookback, frequency='30T')).values
        volume = fill(data.history(coin, 'volume',
                              bar_count=lookback, frequency='30T')).values

        # close[-1] is the last value in the set, which is the equivalent
        # to current price (as in the most recent value)
        # displays the minute price for each pair every 30 minutes
        print('{now}: {pair} -\tO:{o},\tH:{h},\tL:{c},\tC:{c},\tV:{v},\tCV:{v2}'.format(
                now=now,
                pair=pair,
                o=opened[-1],
                h=high[-1],
                l=low[-1],
                c=close[-1],
                v=volume[-1],
                v2=current_volume,
             ))

        # -------------------------------------------------------------
        # --------------- Insert Your Strategy Here -------------------
        # -------------------------------------------------------------

def analyze(context=None, results=None): pass

Get the universe for a given exchange and a given base_currency market

Example: Poloniex BTC Market

def universe(context, lookback_date, current_date):

get all the pairs for the given exchange and given market

context.coins = context.exchanges[context.exchange].assets
context.coins = [c for c in context.coins if c.quote_currency == context.base_currency]

return context.coins

Replace all NA, NAN or infinite values with its nearest value

def fill(series): if isinstance(series, pd.Series): return series.replace([np.inf, -np.inf], np.nan).ffill().bfill() elif isinstance(series, np.ndarray): return pd.Series(series).replace( [np.inf, -np.inf], np.nan ).ffill().bfill().values else: return series

if name == 'main': parser = argparse.ArgumentParser()

# Directory Parameters:
parser.add_argument('--exchange', '-x', type=str, default='poloniex',
                    help='Specify exchange')

start_date = pd.to_datetime('2018-01-01', utc=True)
end_date = pd.to_datetime('2018-01-22', utc=True)

args = parser.parse_args()
performance = run_algorithm(# start=start_date, end=end_date,
                            capital_base=1.0,  # amount of base_currency
                            initialize=initialize,
                            handle_data=handle_data,
                            analyze=analyze,
                            exchange_name=args.exchange,
                            data_frequency='minute',
                            base_currency='btc',
                            live_graph=False,
                            live=True,
                            simulate_orders=True,
                            algo_namespace='simple_universe')


# Anything else?

Trying to get all of these 3 exchanges to run paper trading. The above script is similar to simple_universe.py and will serve to test other exchange data are retrieved properly and compatible with Catalyst. Currently, only Binance (out of binance, kucoin, cryptopia) are working out of the box, in terms of providing me with history data. 

Sincerely,
`avn3r`
fredfortier commented 6 years ago

Thanks for that. I'm aware of the issue with Cryptopia. We're working on releasing something currently on a separate branch for later this week. This should solve this problem as well as offer more historical data. We'll validate through your script before releasing.

On Mon, Jan 22, 2018 at 7:43 PM Abner Ayala-Acevedo < notifications@github.com> wrote:

Dear Catalyst Team, Environment

  • Operating System: Linux Conda environment with python 2.7, pip install enigma-catalyst

Description of Issue:

1.

Current Volume is displaying the volume in the market currency instead of the base currency. For example, eth_btc reports current_volume= data.current(coin, 'volume') in eth volume instead of BTC volume. Inconsistent with data.history API which uses base currency for all values.

  • What did you expect to happen? current daily/24h volume in BTC

    • What happened instead? got current daily/24h volume in ETH (market currency) 2.

    CCXT Kucoin: Many request error when getting the history of the pairs. Error: ccxt.base.errors.RequestTimeout: kucoin GET https://api.kucoin.com/v1/open/chart/history?from=1516062779&symbol=CAT-BTC&type=30&resolution=30&to=1516667579 request timeout 3.

    CCXT Cryptopia: Cannot get all BTC pairs. Mainly because some pair symbols are all numbers and it it converting the symbol into an int instead of a string. Error: AttributeError: 'int' object has no attribute 'lower' catalyst/exchange/ccxt/ccxt_exchange.py", line 534, Line 534 creates the error.

  • What did you expect to happen?

  • What happened instead?

Reproduction Steps

  1. Run code using: python -x

from datetime import timedelta

import numpy as np import pandas as pd import argparse

from catalyst import run_algorithm from catalyst.exchange.utils.exchange_utils import get_exchange_symbols from catalyst.api import (symbols, )

def initialize(context): context.i = -1 # minute counter context.exchange = context.exchanges.values()[0].name.lower() context.base_currency = context.exchanges.values()[0].base_currency.lower()

print(context.coins)

def handle_data(context, data): context.i += 1 lookback_days = 7 # 7 days

# current date & time in each iteration formatted into a string
now = pd.to_datetime(data.current_dt, utc=True)
date, time = now.strftime('%Y-%m-%d %H:%M:%S').split(' ')
lookback_date = now - timedelta(days=lookback_days)

one_day_in_minutes = 1440  # 60 * 24 assumes data_frequency='minute'
# update universe everyday at midnight
if not context.i % one_day_in_minutes:
    context.universe = universe(context, lookback_date, now)

# get data every 30 minutes
minutes = 30
# get lookback_days of history data: that is 'lookback' number of bins
lookback = one_day_in_minutes / minutes * lookback_days
if not context.i % minutes and context.universe:
    # we iterate for every pair in the current universe
    for coin in context.coins:
        pair = str(coin.symbol)

        # Get 30 minute interval OHLCV data. This is the standard data
        # required for candlestick or indicators/signals. Return Pandas
        # DataFrames. 30T means 30-minute re-sampling of one minute data.
        # Adjust it to your desired time interval as needed.
        current_volume = data.current(coin, 'volume')

        opened = fill(data.history(coin, 'open',
                              bar_count=lookback, frequency='30T')).values
        high = fill(data.history(coin, 'high',
                              bar_count=lookback, frequency='30T')).values
        low = fill(data.history(coin, 'low',
                              bar_count=lookback, frequency='30T')).values
        close = fill(data.history(coin, 'price',
                              bar_count=lookback, frequency='30T')).values
        volume = fill(data.history(coin, 'volume',
                              bar_count=lookback, frequency='30T')).values

        # close[-1] is the last value in the set, which is the equivalent
        # to current price (as in the most recent value)
        # displays the minute price for each pair every 30 minutes
        print('{now}: {pair} -\tO:{o},\tH:{h},\tL:{c},\tC:{c},\tV:{v},\tCV:{v2}'.format(
                now=now,
                pair=pair,
                o=opened[-1],
                h=high[-1],
                l=low[-1],
                c=close[-1],
                v=volume[-1],
                v2=current_volume,
             ))

        # -------------------------------------------------------------
        # --------------- Insert Your Strategy Here -------------------
        # -------------------------------------------------------------

def analyze(context=None, results=None): pass

Get the universe for a given exchange and a given base_currency market

Example: Poloniex BTC Market

def universe(context, lookback_date, current_date):

get all the pairs for the given exchange and given market

context.coins = context.exchanges[context.exchange].assets
context.coins = [c for c in context.coins if c.quote_currency == context.base_currency]

return context.coins

Replace all NA, NAN or infinite values with its nearest value

def fill(series): if isinstance(series, pd.Series): return series.replace([np.inf, -np.inf], np.nan).ffill().bfill() elif isinstance(series, np.ndarray): return pd.Series(series).replace( [np.inf, -np.inf], np.nan ).ffill().bfill().values else: return series

if name == 'main': parser = argparse.ArgumentParser()

# Directory Parameters:
parser.add_argument('--exchange', '-x', type=str, default='poloniex',
                    help='Specify exchange')

start_date = pd.to_datetime('2018-01-01', utc=True)
end_date = pd.to_datetime('2018-01-22', utc=True)

args = parser.parse_args()
performance = run_algorithm(# start=start_date, end=end_date,
                            capital_base=1.0,  # amount of base_currency
                            initialize=initialize,
                            handle_data=handle_data,
                            analyze=analyze,
                            exchange_name=args.exchange,
                            data_frequency='minute',
                            base_currency='btc',
                            live_graph=False,
                            live=True,
                            simulate_orders=True,
                            algo_namespace='simple_universe')

Anything else?

Trying to get all of these 3 exchanges to run paper trading. The above script is similar to simple_universe.py and will serve to test other exchange data are retrieved properly and compatible with Catalyst. Currently, only Binance (out of binance, kucoin, cryptopia) are working out of the box, in terms of providing me with history data.

Sincerely, avn3r

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/182, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-QnRjSQTpHoR_prWL02gQ-bFIaF9eks5tNSs-gaJpZM4Ro4nf .

avn3r commented 6 years ago

Any Updates on this bug? Still unable to get Cryptopia to paper trade.

fredfortier commented 6 years ago

Expect something this week for sure On Fri, Feb 23, 2018 at 4:50 PM Abner Ayala-Acevedo < notifications@github.com> wrote:

Any Updates on this bug? Still unable to get Cryptopia to paper trade.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/182#issuecomment-368180814, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-QqO9CXHXr79zCh-eTWZnIPp5mzoaks5tX1zFgaJpZM4Ro4nf .

fredfortier commented 6 years ago

Actively testing this at the moment. This took longer than expected due to complexity of the data. Using CCXT as a common interface help, but we had quite a few edge cases to deal with.

gitlines commented 6 years ago

Since you you use ccxt (which is not documented anywhere) should it be possible to ingest data from a ccxt exchange besides beside polo, bitfinex and bittrex? Trying for example: catalyst ingest-exchange -x binance -i btc_usdt does not work. Thank you

lenak25 commented 6 years ago

Hi @gitlines, catalyst is indeed using CCXT which is also mentioned in our documentation. We are still actively working to add more historical data (like binance) to our servers, to enable users to ingest it and backtest their strategies with. Unfortunately it takes much longer than anticipated but hopefully we will be able to release it soon.