update cchardet to faust-cchardet maintained fork

shimondoodkin commented 1 year ago

it did not build with python3.11

bmoscon commented 1 year ago

probably would be better to just remove the dependency? Does using either of these make a difference in python 3.11 - i.e. do you see a performance degradation without cchardet vs with?

shimondoodkin commented 1 year ago

Sorry I don't have information on that. I don't have a clue where it is used.

bmoscon commented 1 year ago

then let's just remove the requirement. If people want to install it themselves they can, its not a real requirement

shimondoodkin commented 1 year ago

I looked into a bit. https://github.com/aio-libs/aiohttp/blob/fecbe999c7a110fbeba8aa6ba269497435b2870d/aiohttp/client_reqrep.py#L997-L1021 looks like if there is no content type mime in response. if it is not a json mime, which is rare . it will hit it every time. removing it will apply a 18ms penalty per request vs 3ms with cchardet which is significant.

so it is for poorly programmed rare exchanges.

that it might add speed to some rare exchanges that dont send content type mime as part of the responses.

performance comparison of charset-normalizer vs cchardet.

cchardet on average is about 5.6 times faster than charset-normalizer.

------------------------------
--> Chardet Conclusions
   --> Avg: 0.0031594255319148936s
   --> 99th: 0.02425s
   --> 95th: 0.00461s
   --> 50th: 0.00037s
------------------------------
--> Charset-Normalizer Conclusions
   --> Avg: 0.017847297872340427s
   --> 99th: 0.19381s
   --> 95th: 0.09307s
   --> 50th: 0.00712s
------------------------------
--> Charset-Normalizer / Chardet: Performance Сomparison
   --> Avg: x0.18 /  x5.55
   --> 99th: x0.13 / x7.69
   --> 95th: x0.05 / x20
   --> 50th: x0.05 / x20

shimondoodkin commented 1 year ago

it is not used at all

from cryptofeed.exchanges import EXCHANGE_MAP
import requests

for a in iter(EXCHANGE_MAP):
    ex=EXCHANGE_MAP[a]
    if len(ex.rest_endpoints)==0:
        continue;

    e=ex.rest_endpoints[0]
    if isinstance(e.routes.instruments, str):
        u=e.address+e.routes.instruments
        print("")
        print(u)
        try:
            response = requests.get(u)
            print(response.status_code)
            print(response.headers['Content-Type'])
        except:
            print('request error')

    else:
        for a in e.routes.instruments:
            u=e.address+a
            print("")
            print(u)
            try:
                response = requests.get(u)
                print(response.status_code)
                print(response.headers['Content-Type'])
            except:
                print('request error')

output:

https://ascendex.com/api/pro/v1/products 200 application/json

https://ascendex.com/api/pro/v2/futures/contract 200 application/json

https://api.bequant.io/api/2/public/symbol 200 application/json

https://dapi.binance.com/dapi/v1/exchangeInfo 200 application/json

https://fapi.binance.com/fapi/v1/exchangeInfo 200 application/json

https://api.binance.us/api/v3/exchangeInfo 200 application/json;charset=UTF-8

https://api.binance.com/api/v3/exchangeInfo 200 application/json;charset=UTF-8

https://api.fmfw.io/api/3/public/symbol 200 application/json

https://spot-api.bit.com/spot/v1/instruments 200 application/json

https://api-pub.bitfinex.com/v2/conf/pub:list:pair:exchange 200 application/json; charset=utf-8

https://api-pub.bitfinex.com/v2/conf/pub:list:currency 200 application/json; charset=utf-8