Kucoin Too Many Requests

[mablue]

I used:

exchange = ccxt.kucoin({
    'enableRateLimit': True,
    ...

On one line one core run of fetch_olhcv loop But still have:

ccxt.base.errors.RateLimitExceeded: kucoin GET https://openapi-v2.kucoin.com/api/v1/market/candles?symbol=ALGO-USDT&type=1hour&startAt=1633301197&endAt=1634741197 429 Too Many Requests {"code":"429000","msg":"Too Many Requests"}

[kroitor]

We need more info on the usage – are you recreating or relaunching that script over and over? Cause that will break this rule:

• https://github.com/ccxt/ccxt/wiki/Manual#notes-on-rate-limiter

[kroitor]

If possible, please paste your complete code to reproduce it:

• https://github.com/ccxt/ccxt/wiki/FAQ#what-is-required-to-get-help

[mablue]

We need more info on the usage – are you recreating or relaunching that script over and over? Cause that will break this rule:

• https://github.com/ccxt/ccxt/wiki/Manual#notes-on-rate-limiter

No one statement on a for loop thats it, ok I will paste here:

import ccxt
exchange = ccxt.kucoin({
    'enableRateLimit': True,
    'apiKey': '',
    'secret': '',
    'password': ''
})
markets = exchange.load_markets()
tf='5m'
while True:
    random.shuffle(markets)
    for market in markets[:50]:
        olhcv = exchange.fetch_ohlcv(symbol=market, timeframe=tf, limit=400)
        print(olhcv)

[krychla1]

Please log your kucoin requests and you will find out that the rate limiting in ccxt works well.

This is cloudflare error on the part of kucoin, which has nothing to do with your own actual requests rate, rather with their overall system health. You should contact them to find out more.

[mablue]

maybe it happend cuz of my location ip ( iran! )

[kroitor]

@mablue yes, that may be the case. I can't reproduce it on my side, with and without proxies. Will try more.

[mablue]

@mablue yes, that may be the case. I can't reproduce it on my side, with and without proxies. Will try more.

Any updates?! I use trmux emulator

[krychla1]

@mablue yes, that may be the case. I can't reproduce it on my side, with and without proxies. Will try more.

its hard to reproduce, it happens time to time, I suppose worldwide during higher traffic. You will not be able to cause it by high load just on your side, unless you have king limits (thousands/sec) or have the possibility to ddos the domain :) I noticed several mentions on reddits with a few similarities, also I face this issue several times/day. I even raised my limits at kucoin, it had no impact.

[kroitor]

@krychla1 thx for your feedback! I'll try on my side to see if tweaking the rate limits helps... Sometimes, their own API docs may be wrong on the actual limits, or might have slightly outdated numbers. Will do my best to investigate it as soon as I can.

[krychla1]

You are welcome, I help if I can.

Limits I confirmed with their agents last week: orderbooks(lvl2, lvl3), the standard limit is 10req/3s their cloudflare limit: 4000/s for a specific endpoint for all users

[mablue]

Thanks I cant send 3 in one sec I think 🤔 Its more problematic on fetch_olhcv as well If you need i can make a vpn server on my pc :)) in iran I have evil possibilities on iran :))

[qkum]

This is how my error looks like:

2021-10-30 13:03:54,919 - freqtrade.exchange.common - WARNING - _async_get_candle_history() returned exception: "kucoin GET https://openapi-v2.kucoin.com/api/v1/market/candles?symbol=FKX-USDT&type=1hour&startAt=1633799034&endAt=1635599034 429 Too Many Requests {"code":"429000","msg":"Too Many Requests"}"

I get it all the time with KuCoin, but not on Binance.

Rate limit 200.

I get this error both on my VM in Germany and my home pc in another EU country.

If I download candle data or just turn dry-run on with the freqtrade.io bot it is spammed until everything is downloaded after 1000 years.

I hope my input is useful.

[qkum]

You are welcome, I help if I can.

Limits I confirmed with their agents last week: orderbooks(lvl2, lvl3), the standard limit is 10req/3s their cloudflare limit: 4000/s for a specific endpoint for all users

And that is in rate limit terms? 4000 instead of standard 200?

[frosty00]

@qkum the default rate limit for kucoin is 333 (three reqs/sec), have you tried using that instead?

[qkum]

@qkum the default rate limit for kucoin is 333 (three reqs/sec), have you tried using that instead?

Thanks for the answer.

I tried with 4000 and it did not change anything.

I will try with 300 then.

[qkum]

Rate limit 300 changes nothing.

I think I will try with 100.

But their API seems to bug in more than one way.

Also like this:

2021-10-30 22:17:03,994 - freqtrade.strategy.interface - WARNING - Outdated history for pair KONO/USDT. Last tick is 17 minutes old
2021-10-30 22:17:05,916 - freqtrade.strategy.interface - WARNING - Outdated history for pair KONO/USDT. Last tick is 17 minutes old
2021-10-30 22:17:08,052 - freqtrade.strategy.interface - WARNING - Outdated history for pair KONO/USDT. Last tick is 17 minutes old
2021-10-30 22:17:09,916 - freqtrade.strategy.interface - WARNING - Outdated history for pair KONO/USDT. Last tick is 17 minutes old

Running a 5 min strategy and 1 or 2 of the 25 pairs not beeing updated with new price data for 17 min is really bad.

When i run a 1hour stategy it does the same with 1 or 2 pairs...not updated for 133 minutes like lol.

If i run a manuel price data download for the bugged pairs there is no problem and the last 3 candles are downloaded asap..so messed up.

Using the freqtrade.io bot.

Again, no problem at all if i run the same dry-run on Binance. Only problems with KuCoin.

[qkum]

@qkum the default rate limit for kucoin is 333 (three reqs/sec), have you tried using that instead?

I found the solution.

Outdated history for pair KONO/USDT. Last tick is 17 minutes old is in reality a bad wording for there have been no trades on this pair for 17 min

I asked the people behind the freqtrade bot to change the wording because it is not logical what it means.

The bot is getting candle data and trading even when it spams what seems like a warning telling you it is missing data. But it is because there have been no trades, so there is no candle data to get from the API.

And I checked the chart manually and there had not been any trades in the time span the bot was warning about.

KuCoin got many pairs with no trades for up to 2+ hours we know now. Something I thought was impossible on the world's 2'd biggest exchange.

[xmatthias]

freqtrade users are seeing this issue as well (https://github.com/freqtrade/freqtrade/issues/5700) - if you look through the messages there, you'll see that several people contacted support about this issue - and got told to "retry immediately - it's our fault - but only with 429000 errors".

Now i have a problem with that statement personally, as i don't see it as official (otherwise their docs would state the same) - but it doesn't feel like it's getting better, but worse at the moment.

a simple script to test/retry this is the following - left it running for 5-10 minutes and ran into it twice.

import ccxt
from datetime import datetime, timezone
from time import sleep

exchange = ccxt.kucoin()
_ = exchange.load_markets()
while True:
    x = exchange.fetch_ohlcv('BTC/USDT', '5m', limit=1000)
    print(f"{datetime.now()} {len(x)} {datetime.fromtimestamp(x[0][0]// 1000, tz=timezone.utc)}")
    sleep(1)

2021-11-03 19:55:28.769703 1000 2021-10-31 07:40:00+00:00          
---------------------------------------------------------------------------                                                            
HTTPError                                 Traceback (most recent call last)                                                            
~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in fetch(self, url, method, headers, body
)                                                                                                                                      
    612             self.logger.debug("%s %s, Response: %s %s %s", method, url, http_status_code, headers, http_response)

--> 613             response.raise_for_status()                                                                                            614      

~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/requests/models.py in raise_for_status(self)                       952         if http_error_msg:                                 
--> 953             raise HTTPError(http_error_msg, response=self)                                                                     
    954                                                            

HTTPError: 429 Client Error: Too Many Requests for url: https://openapi-v2.kucoin.com/api/v1/market/candles?symbol=BTC-USDT&type=5min&s
tartAt=1635665729&endAt=1635965729                                                                                                                                                                        
During handling of the above exception, another exception occurred:                                                                    

RateLimitExceeded                         Traceback (most recent call last)                                                            
<ipython-input-8-3a90f43b9c1c> in <module>                         
      1 while True:                                                
----> 2     x = exchange.fetch_ohlcv('BTC/USDT', '5m', limit=1000) 
      3     print(f"{datetime.now()} {len(x)} {datetime.fromtimestamp(x[0][0]// 1000, tz=timezone.utc)}")                                    4     sleep(1)                                                                                                                   
      5                          

~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/kucoin.py in fetch_ohlcv(self, symbol, timeframe, since, l
imit, params)               
    876             request['startAt'] = int(int(math.floor(since / 1000)))                                                                877         request['endAt'] = int(int(math.floor(endAt / 1000)))                                                                  
--> 878         response = self.publicGetMarketCandles(self.extend(request, params))                                                   
    879         #                                                  
    880         #     {                                            

~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in inner(_self, params, context)         
    455                 if context is not None:                                                                                        
    456                     inner_kwargs['context'] = params                                                                           
--> 457                 return entry(_self, **inner_kwargs)        
    458             return inner                                   
    459         to_bind = partialer()                              

~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in request(self, path, api, method, param
s, headers, body, config, context)                                
    504     def request(self, path, api='public', method='GET', params={}, headers=None, body=None, config={}, context={}):
    505         """Exchange.request is the entry point for all generated methods"""                                                    
--> 506         return self.fetch2(path, api, method, params, headers, body, config, context)                                          
    507                                                            
    508     @staticmethod                                          

~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in fetch2(self, path, api, method, params
, headers, body, config, context)                        
    500         self.lastRestRequestTimestamp = self.milliseconds()
    501         request = self.sign(path, api, method, params, headers, body)                                                          
--> 502         return self.fetch(request['url'], request['method'], request['headers'], request['body']) 
    503                                                                                                                        [0/2005]
    504     def request(self, path, api='public', method='GET', params={}, headers=None, body=None, config={}, context={}):            

~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in fetch(self, url, method, headers, body)            
    628             details = ' '.join([self.id, method, url])     
    629             self.handle_errors(http_status_code, http_status_text, url, method, headers, http_response, json_response, request_headers, request_body)                                             
--> 630             self.handle_http_status_code(http_status_code, http_status_text, url, method, http_response)                       
    631             raise ExchangeError(details) from e            
    632                          

~/.pyenv/versions/3.9.7/envs/trade_397_copy/lib/python3.9/site-packages/ccxt/base/exchange.py in handle_http_status_code(self, http_status_code, http_status_text, url, method, body)                     
    664         if string_code in self.httpExceptions:                                                                                 
    665             Exception = self.httpExceptions[string_code]   
--> 666             raise Exception(' '.join([self.id, method, url, string_code, http_status_text, body]))                             
    667                                                            
    668     def parse_json(self, http_response):                   

RateLimitExceeded: kucoin GET https://openapi-v2.kucoin.com/api/v1/market/candles?symbol=BTC-USDT&type=5min&startAt=1635665729&endAt=1635965729 429 Too Many Requests {"code":"429000","msg":"Too Many Requests"}    

I think i can work around it by using the exchange.last_json_response object to get the "real" code - but maybe this should not raise a RateLimitExceeded (even though the http status code says so) - but rather a different exception, clearly identifying it as "retry immediately" request.

[Theagainmen]

I contacted the API developers of KuCoin and they are aware of this issue. They will release a new high performance endpoint to solve this around next week. I assume CCXT needs to implement this new high performance endpoint then? I'm also a freqtrade user, and I see @xmatthias is active in here so I'll just tag him so he knows this too :)

Quoted from developer:

Next week we will release a new endpoint with higher performance.

The release is a new endpoint. the original endpoint still exists The usage is basically the same, and the data cache will be used, so the performance is higher.

[xmatthias]

i guess they could also just pay the higher bill on cloudflare ... :laughing: but we'll see what they'll come up with ...

[mablue]

@kucoin can not become a Bill Gates, Unless you pay higher bills to get more gates.

[muratbulat]

I lost half day, finally a correct answer! Thanks.

[joeysimon3]

When is this going to be fixed? Does anyone know a work around?

[b-zig]

When is this going to be fixed? Does anyone know a work around?

looks like Kucoin has released a high performance API to address rate limiting, however they're monitoring it for another few days before updating the API documentation with the new server. I got this from their telegram API support group/

[joeysimon3]

When is this going to be fixed? Does anyone know a work around?

looks like Kucoin has released a high performance API to address rate limiting, however they're monitoring it for another few days before updating the API documentation with the new server. I got this from their telegram API support group/

Thank you! After this new API is implemented, how long until it will be able to be used on ccxt?

[rapmd73]

I am also experiencing this issue, and have set my rate limit as high as 5,000. I am using the version 2 API, and have not found a way around this issue.

Is this issue confirmed to be on kucoins side?

Thank you in advance.

[kroitor]

Hi everyone!

This issue is on the exchange side. The problem is that they don't document the actual rate limits on the ohlcv endpoint and many other endpoints and they don't serve the usage stats, which makes it impossible to fix on the CCXT side.

The only meaningful workaround solution for now is to add the handling for RateLimitExceeded like shown in the following examples:

• https://github.com/ccxt/ccxt/blob/master/examples/py/kucoin-rate-limit.py
• https://github.com/ccxt/ccxt/blob/master/examples/py/async-kucoin-rate-limit.py

# Python sync

import ccxt

print('CCXT Version:', ccxt.__version__)

exchange = ccxt.kucoin()
markets = exchange.load_markets()
i = 0
while True:
    try:
        symbol = 'BTC/USDT'
        timeframe = '5m'
        since = None
        limit = 1000
        ohlcvs = exchange.fetch_ohlcv(symbol, timeframe, since, limit)
        now = exchange.milliseconds()
        datetime = exchange.iso8601(now)
        print(datetime, i, 'fetched', len(ohlcvs), symbol, timeframe, 'candles',
            'from', exchange.iso8601(ohlcvs[0][0]),
            'to', exchange.iso8601(ohlcvs[len(ohlcvs)-1][0]))
    except ccxt.RateLimitExceeded as e:
        now = exchange.milliseconds()
        datetime = exchange.iso8601(now)
        print(datetime, i, type(e).__name__, str(e))
        exchange.sleep(10000)
    except Exception as e:
        print(type(e).__name__, str(e))
        raise e
    i += 1

# Python async

from asyncio import run
import ccxt.async_support as ccxt  # noqa: E402

print('CCXT Version:', ccxt.__version__)

async def main():
    exchange = ccxt.kucoin()
    markets = await exchange.load_markets()
    i = 0
    while True:
        try:
            symbol = 'BTC/USDT'
            timeframe = '5m'
            since = None
            limit = 1000
            ohlcvs = await exchange.fetch_ohlcv(symbol, timeframe, since, limit)
            now = exchange.milliseconds()
            datetime = exchange.iso8601(now)
            print(datetime, i, 'fetched', len(ohlcvs), symbol, timeframe, 'candles',
                'from', exchange.iso8601(ohlcvs[0][0]),
                'to', exchange.iso8601(ohlcvs[len(ohlcvs)-1][0]))
        except ccxt.RateLimitExceeded as e:
            now = exchange.milliseconds()
            datetime = exchange.iso8601(now)
            print(datetime, i, type(e).__name__, str(e))
            await exchange.sleep(10000)
        except Exception as e:
            print(type(e).__name__, str(e))
            raise e
        i += 1

run (main())

[dawadam]

If it's not too much to ask, is it possible to have the solution in javascript?

[kroitor]

@dawadam will post a link here.

[kroitor]

@dawadam

• https://github.com/ccxt/ccxt/blob/master/examples/js/kucoin-rate-limit.js

"use strict";

const ccxt = require ('ccxt')

async function main () {

    const exchange = new ccxt.kucoin()
    const markets = await exchange.loadMarkets ()
    const timeframe = '5m'
    const symbol = 'BTC/USDT'
    const since = undefined
    const limit = 1000

    let i = 0
    while (true) {
        try {
            const ohlcvs = await exchange.fetchOHLCV(symbol, timeframe, since, limit)
            const now = exchange.milliseconds()
            const datetime = exchange.iso8601(now)
            console.log(datetime, i, 'fetched', ohlcvs.length, symbol, timeframe, 'candles',
                'from', exchange.iso8601(ohlcvs[0][0]),
                'to', exchange.iso8601(ohlcvs[ohlcvs.length-1][0]))
        } catch (e) {
            if (e instanceof ccxt.RateLimitExceeded) {
                const now = exchange.milliseconds()
                const datetime = exchange.iso8601(now)
                console.log(datetime, i, e.constructor.name, e.message)
                await exchange.sleep(10000)
            } else {
                console.log(e.constructor.name, e.message)
                throw e
            }
        }
        i += 1
    }
}

main ()

[ttodua]

@kroitor I think to address this specific issue, you did all (the posted examples) whatever was doable in current situation (until KuCoin releases updated engine, which will be a different PR). I think this issue can be closed for now?

[weerdenburg]

Is the new kuCoin API endpoint already in use by now? And should I do something special in ccxt to being able to use that new endpoint? Because we are still having the rate limit issues now and then...

[rapmd73]

I doubt there is going to be any change in kucoin's rate limiting issues simply because of their own comments within regard to server support.

They're on technical support department says that the only way to truly resolve the problem is for you to have more IP addresses that you can rotate between.

I personally use a sliding scale when I get the response to slow things down slightly based upon the number of responses. I have found that it's not too much of an issue as long as you don't get too aggressive with the exchange. You can easily use around 500 millisecond rate limits as a base value and then add upon that based upon the number of messages per second to get for rate limiting.

[Homeyli]

Thanks I cant send 3 in one sec I think thinking Its more problematic on fetch_olhcv as well If you need i can make a vpn server on my pc :)) in iran I have evil possibilities on iran :))

داداش همین که فهمیدن از ایران ریکوست میدی یه تف نکردند تو صورتت شانس اوردی