Binance sideloading API

[AlexTheLion123]

Goal

To find the easiest to use and clearest API for sideloading Binance data.

Notes

• We have generate_pair_for_binance_data so user doesn't have to manually create TradingPairIdentifier. Similarly for generate_exchange_for_binance_data
• Feels a bit wrong to make user use add_info_columns and overwrite_cached_data themselves

Current API Summary

The following represents the current API for sideloading Binance data. See PR #664 for reference and for the notebook. The notebook currently follows six main steps:

1. OHLCV data

downloader = BinanceDownloader()

pair = generate_pair_for_binance_data(BASE_TOKEN_SYMBOL, QUOTE_TOKEN_SYMBOL, PAIR_FEE)

symbol = "ETHUSDT"

# use stop_loss_time_bucket since, in this case, it's more granular data than the candle_time_bucket
# we later resample to the higher time bucket for the backtest candles
df = downloader.fetch_candlestick_data(
    symbol,
    STOP_LOSS_TIME_BUCKET,
    START_AT_DATA,
    END_AT,
)

df = add_info_columns_to_ohlc(df, pair)

# we need to overwrite the cache for when the candles are loaded later
downloader.overwrite_cached_data(df, symbol, STOP_LOSS_TIME_BUCKET, START_AT_DATA, END_AT)

2. Candle universe, exchange universe, and pairs_df

path = downloader.get_parquet_path(symbol, STOP_LOSS_TIME_BUCKET, START_AT_DATA, END_AT)

candle_universe, stop_loss_candle_universe = load_candle_universe_from_parquet(
    pair=pair,
    file=path,
    include_as_trigger_signal=True,
    resample=CANDLE_TIME_BUCKET, 
)

binance_exchange = generate_exchange_for_binance_data(pair)

exchange_universe = ExchangeUniverse.from_collection([binance_exchange])

# TODO how to do for multiple pairs?
pairs_df = candle_universe.pairs.first().reset_index()

3. Lending reserve universe

client = Client.create_jupyter_client()
universe = client.fetch_lending_reserve_universe()

usdc_desc = (ChainId.polygon, LendingProtocolType.aave_v3, "USDC")
weth_desc = (ChainId.polygon, LendingProtocolType.aave_v3, "WETH")

lending_reserve_universe = universe.limit([usdc_desc, weth_desc])

4. Lending and supply data

eth_reserve_id = lending_reserve_universe.resolve_lending_reserve(weth_desc).reserve_id
usdc_reserve_id = lending_reserve_universe.resolve_lending_reserve(usdc_desc).reserve_id

eth_lending_data = downloader.fetch_lending_rates("ETH", CANDLE_TIME_BUCKET, START_AT_DATA, END_AT)
eth_supply_data = convert_binance_lending_rates_to_supply(eth_lending_data)

usdc_lending_data = downloader.fetch_lending_rates("USDC", CANDLE_TIME_BUCKET, START_AT_DATA, END_AT)
usdc_supply_data = convert_binance_lending_rates_to_supply(usdc_lending_data)

5. Lending candle universe

lending_candle_type_map = convert_interest_rates_to_lending_candle_type_map({"reserve_id":eth_reserve_id,"lending_data": eth_lending_data, "supply_data":eth_supply_data}, {"reserve_id":usdc_reserve_id,"lending_data": usdc_lending_data, "supply_data":usdc_supply_data})

lending_candle_universe = LendingCandleUniverse(lending_candle_type_map, lending_reserve_universe)

6. Complete dataset

dataset = Dataset(
    time_bucket=CANDLE_TIME_BUCKET,
    exchanges=exchange_universe,
    pairs=pairs_df,
    candles=candle_universe.df,
    backtest_stop_loss_time_bucket=STOP_LOSS_TIME_BUCKET,
    backtest_stop_loss_candles=candle_universe.df,
    lending_candles=lending_candle_universe,
    lending_reserves=lending_reserve_universe,
)

[miohtama]

• downloader.overwrite_cached_data(df, symbol, STOP_LOSS_TIME_BUCKET, START_AT_DATA, END_AT) -> input and output format should stay same, if needed write another file
• add_info_columns_to_ohlc(df, pair) -> must support multiple pairs -> see https://github.com/tradingstrategy-ai/trading-strategy/blob/master/scripts/fetch-binance-candles.py (don't mind the merge conflict)

client = Client.create_jupyter_client()
universe = client.fetch_lending_reserve_universe()

usdc_desc = (ChainId.polygon, LendingProtocolType.aave_v3, "USDC")
weth_desc = (ChainId.polygon, LendingProtocolType.aave_v3, "WETH")

lending_reserve_universe = universe.limit([usdc_desc, weth_desc])

-> We do not need to fetch anything from the Trading Strategy oracle, because we are using Binance data. All data should be derived from Binance ticket symbols.

binance_exchange = generate_exchange_for_binance_data(pair)

-> The exchange does not need to know anything about it's pair or a pair. This information is stored in the pair universe.

[miohtama]

Here is some example function signatures

def load_binance_data(
      symbols: List[str],
      interest=False,
) -> Dataset:
     """End-user facing easy function. Does 'everything'.

      :param symbols:
            ["ETHUSDT", "BTCUSDT", ....]
      """
    # create downloader
    # check existing cache files (one for lending, one for OHLCV)
    # if cache file exist open it
    # otherwise write new raw binance data cache file
    # with raw binance data
    # - construct "binance" exchange
    # - map symbols strs to PandasPairUniverse
   # - convert candles 
   # - convert lending data
   # - wrap as Dataset object

def create_binance_universe(      
     symbols: List[str],
     interest=False,
     reserve_currency_symbol="USDT",
) -> TradingStrategyUniverse:
   """Load Binance data and create Trading Strategy Universe"""
   # - load_binance_data() 
   # - create AssetIdentifier for reserve_currency_symbol
   # - Pass dataset TradingStrategyUniverse.create_from_dataset()

# ...a lot of helper and internal functions go here...

[AlexTheLion123]

Updated API, now to put it all under 1 function:

downloader = BinanceDownloader()

pair = generate_pair_for_binance(BASE_TOKEN_SYMBOL, QUOTE_TOKEN_SYMBOL, PAIR_FEE, 12398437)

symbol = "ETHUSDT"

# use stop_loss_time_bucket since, in this case, it's more granular data than the candle_time_bucket
# we later resample to the higher time bucket for the backtest candles
df = downloader.fetch_candlestick_data(
    symbol,
    STOP_LOSS_TIME_BUCKET,
    START_AT_DATA,
    END_AT,
)

candle_df = add_info_columns_to_ohlc(df, {symbol:pair})

candle_universe, stop_loss_candle_universe = load_candle_universe_from_dataframe(
    pair=pair,
    df=candle_df,
    include_as_trigger_signal=True,
    resample=CANDLE_TIME_BUCKET, 
)

exchange_universe = generate_exchange_universe_for_binance(pair_count=1)

pairs_df = candle_universe.get_pairs_df()

usdc_reserve = generate_lending_reserve_for_binance(pair.quote.token_symbol, pair.quote.address, 1)
weth_reserve = generate_lending_reserve_for_binance(pair.base.token_symbol, pair.base.address, 2)

lending_reserve_universe = LendingReserveUniverse({usdc_reserve.reserve_id: usdc_reserve, weth_reserve.reserve_id: weth_reserve})

lending_candle_type_map = downloader.load_lending_candle_type_map({weth_reserve.reserve_id: "ETH", usdc_reserve.reserve_id: "USDC"}, CANDLE_TIME_BUCKET, START_AT_DATA, END_AT)

lending_candle_universe = LendingCandleUniverse(lending_candle_type_map, lending_reserve_universe)

dataset = Dataset(
    time_bucket=CANDLE_TIME_BUCKET,
    exchanges=exchange_universe,
    pairs=pairs_df,
    candles=candle_universe.df,
    backtest_stop_loss_time_bucket=STOP_LOSS_TIME_BUCKET,
    backtest_stop_loss_candles=candle_universe.df,
    lending_candles=lending_candle_universe,
    lending_reserves=lending_reserve_universe,
)