Running multiple live coins

[Ericxgao]

I'm running multiple live trading sessions and wanted to know when the pricing data for btc_usd needed to be pulled.

If I have 40 coins trading and all of them try to ingest that daily data, I run into an ccxt.base.errors.DDoSProtection: bitfinex ERR_RATE_LIMIT error - how can I work around this?

I am staggering the initialization of each algorithm, but I'm worried that after midnight they will all trigger this call again and kill the algo.

[fredfortier]

We're pulling the benchmark data from Bitfinex as a temporary solution. While it seem innocuous at first, several people reported issues like this one. We'll fix this shortly to replace by a solution which executes without querying an exchange. We will have this ready in a few days which should allow you to drop your workaround.

On Mon, Jan 29, 2018 at 7:28 PM Ericxgao notifications@github.com wrote:

I'm running multiple live trading sessions and wanted to know when the pricing data for btc_usd needed to be pulled.

If I have 40 coins trading and all of them try to ingest that daily data, I run into an ccxt.base.errors.DDoSProtection: bitfinex ERR_RATE_LIMIT error - how can I work around this?

I am staggering the initialization of each algorithm, but I'm worried that after midnight they will all trigger this call again and kill the algo.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/enigmampc/catalyst/issues/196, or mute the thread https://github.com/notifications/unsubscribe-auth/ABZ-QmWi5pwqzajfozX0vqYdgg_Yq5Jtks5tPmI9gaJpZM4RxlAB .

[fredfortier]

Included this in our next milestone.

[salihkilic]

Pulling live data from the "Simple Universe" example in the docs also results in the following error:

Traceback (most recent call last):
  File "C:\Users\[my_username]\AppData\Local\Programs\Python\Python36\lib\site-packages\ccxt\base\exchange.py", line 356, in fetch
    response.raise_for_status()
  File "C:\Users\[my_username]\AppData\Local\Programs\Python\Python36\lib\site-packages\requests\models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://api.bitfinex.com/v2/candles/trade:15m:tSNGETH/hist?sort=1&limit=202&start=1521477780000

[lenak25]

@Ericxgao , just to make sure I understand you correctly: you reach the rate limit due to the downloaded benchmark data or because you are trading many coins live in Bitfinex? @salihkilic , thanks for this update, we are aware that currently there is a problem in running the "simple universe" example in live mode in Bitfinex, due to their low rate limit. We plan to introduce some improvements to the way data is retrieved to better handle this. In the meantime, you can use the other exchanges for this purpose.

[salihkilic]

@lenak25 I haven't been able to make it work (on whatever exchange) even though I saw other people being able to and I found out what my problem is.

You can feed data.history() a list of coins. So when I do the following:

data.history(context.coins,
                                             ['open', 'high', 'low', 'price', 'volume'],
                                             bar_count=lookback,
                                             frequency=str(minutes) + 'T')

I hit the rate limit automatically, while retrieving the same data in a for loop doesn't. Like the following:

for coin in context.coins:
    data.history(coin,
                                                 ['open', 'high', 'low', 'price', 'volume'],
                                                 bar_count=lookback,
                                                 frequency=str(minutes) + 'T')

Example one is much easier and works perfectly fine in backtesting, just not live. I'm quite the novice programmer, so I'm searching for the declaration of data.history in the library but I can't seem to find it.

Edit: Now that I think of it, it would be excellent if I could start a live trading session by populating my history with longer time in between calls for the full lookback period (as not to hit the rate limit) and then when I have enough data I can do faster api calls for the periods in between handle_data() because they don't have the weight of the initial ones. I'm not sure if the data.history() always requests the exchange for the full data of the whole lookback period or just for the information that's missing locally since the last call.

[lenak25]

Thanks for the feedback @salihkilic , we will check this. Regarding the declaration of the history function, it is located at the Cython _protocol.pyx file, which in turn calls the get_history_window function of the DataPortalExchange class (exchange_data_portal.py) .

[salihkilic]

@lenak25 Thanks for pointing that out! I have some new findings for you. It seems even my new approach hits a rate limit bottleneck whenever I try to download two days worth of 15T data for more than two assets. I've tried looking through the class you pointed out but again, this goes into a level of abstraction I'm far from comfortable with yet. I can't even find where the rate limit is determined or called within that function, if at all.

So even though I'd love to have contributed this to the project myself, my suggestion would be that the rate limit isn't just set by the time between requests, but also considers the weight of the request (which is simply counting the amount of bars we request per asset). By making two optinal arguments in the data.history() function (something like "rate_limit" and "weight_per_bar") we'd enable users to change these settings accordingly to a multitude exchanges that are going to be implemented (with each differently working rate limits). Some exchanges care about absolute time in between requests, others reject requests above a total weight (amount of requests amount of bars amount of assets) within a certain timeframe (binance does this for example).

It would also open the way for users of the platform to share these optimal settings for each exchange and make this rather opaque ratelimit system a bit more clear (and easier to solve for the user!).

[salihkilic]

I've been trying a couple of things the last two hours and it makes even less sense to me now. Right now bitfinex is giving me DDoS alerts when I download requests 4 bars long @ 15T for 4 assets. I have absolutely no clue what is happening anymore. Does anyone have an example using multiple assets that works?

[lenak25]

@salihkilic , as mentioned, Bitfinex has a very low rate limit, meaning it can be challenging to perform many requests to this exchange each frame. When trying to assess the number of requests performed, you should take into consideration the following:

1. A different request is sent per each asset
2. At the current implementation: a different request is sent per each field - we plan to fix it ASAP. If you need more help in trying to understand the behavior of your algorithm, you can attach the code implementing the candles fetch.

[salihkilic]

@lenak25 Oh wow! That second point shines some light on what's happening! Thanks for the reply! I'll take a look if I can use CCXT directly for candle requests in a smarter/lightweight manner in the meantime.

[kooomix]

Can you tell what is the requests frequency allowed for an asset per field?

[lenak25]

@kooomix, the rate limit is exchange specific and can vary between different API calls. To have a sense of the rate limit of the exchange you are using you can have a look at the rateLimit parameter exposed by CCXT which can be viewed using: context.exchanges[<exchange_name>].api.rateLimit