search

ccxt / ccxt

A JavaScript / TypeScript / Python / C# / PHP cryptocurrency trading API with support for more than 100 bitcoin/altcoin exchanges
https://docs.ccxt.com
MIT License
32.44k stars 7.46k forks source link

New Rate-limiting Method #13949

Open DoctorSlimm opened 2 years ago

DoctorSlimm commented 2 years ago

This issue is about new format of rate-limiting requests in ccxt.

Old Method

Previously all endpoints within an exchange had the same weight associated, described by the rateLimit property inside each exchange object. This property is numeric and represents the ms between subsequent requests. So if an exchange has a rateLimit property of 100, this equates to 100ms between individual requests being sent by ccxt on average. This in turn means that there are 1000ms / 100ms = 10 requests per second. Previously all endpoints had the same weight, of none, which translated to a weight of 1 and the same rate-limiting rule defined by the rateLimit property was applied to all endpoints.

In the following old-implentation example all endpoints are ratelimited in the same way, in this case 100ms between requests which translates to 10 requests per second

...
'rateLimit': 100,
...
'public': {
    'get': [
        'market/tickers',
        'market/orderbook',
        ...
    ],
'private': {
    'get': [
        'account/balance',
        ...
    ],
    'post':[
        'trade/order',
        ...
    ],
}

New Method

New rate-limiting method allows different endpoints to be rate-limited differently, since many exchanges apply different rules to different endpoints. For example trading endpoints tend to allow more requests than data-fetching endpoints, and it is often the case that symbol-specific endpoints can be called more often than those which return information for all or multiple symbols. The new method changes the endpoints from a list to a key-value endpoint-weight pairing which can be seen in the exchange classes to be: 

...
'rateLimit': 100,
...
'public': {
    'get': {
        'market/tickers': 2.5,
        'market/orderbook': 2,
        ...
    },
'private': {
    'get': {
        'account/balance': 1.5,
        ...
    },
    'post':{
        'trade/order': 1
        ...
    },
}

The weight attached represents the weight of the endpoint. The rateLimit property will always be the rate-limit of the endpoint which can be called most frequently. All weights of other endpoints will then be re-defined in terms of this cheapest endpoint.

In the preceding case, the actual number of times which an endpoint can be called per second can be calculated by:

1000 / ( rateLimit * weight)

so for the above example, the efective rate-limits for the endpoints would be:

The last endpoint is the one which can be called most frequently, this is the one which is used to calculate the baseline rateLimit property and is done like so:

1000ms / (number of requests allowed per second) so if the documentation states 20 requests per 2 seconds this is effectively 10 requests per second meaning that the rateLimit => 1000 / 10 = 100.

Start with the certified exchanges

and these are some popular exchanges it should also be unified in

And then the rest of the exchanges

Serpens66 commented 2 years ago

Make sure you also keep the different allowed usage per time in mind. Eg. at binance you have (if I understood documentation correct): 6100 per 5 minutes raw calls, 1200 per minute per IP for _api calls 12000 per minute per IP for _sapi calls 180000 for sapi account call like "fiat/orders" while the call itself costs 90000 (for whatever reason) And more.

In my own python code I assigned multiple weight values to the calls for each category, eg: "get_trades":{"raw":1,"_api":10}, So the get_trades call ("myTrades") does consume the raw limit and the _api limit.

dholladay00 commented 3 months ago

Is there a way to query the weight of a given endpoint? How would this look in python?

samgermain commented 3 months ago

Is there a way to query the weight of a given endpoint? How would this look in python?

Currently there isn't, I created a PR for this feature, and we may add it, or may close the PR https://github.com/ccxt/ccxt/pull/22584