mikekonan / exchange-proxy

exchange-proxy
GNU General Public License v3.0
110 stars 32 forks source link

More issues with AgeFilter #35

Closed jurekl closed 9 months ago

jurekl commented 2 years ago

Hi Mike, Unfortunately, it is still not good. I have in the configuration, among others:

                {
                        "method": "VolumePairList",
                        "number_assets": 100,
                        "sort_key": "quoteVolume",
                        "refresh_period": 1800
                },
                {
                       "method": "AgeFilter", "min_days_listed": 31
                },
...
...
...

I just noticed in FreqUI that my whiltelist is only 5 pairs. A quick check in the FT log and we see such wonders:

2022-01-18 21:46:01 - VolumePairList - INFO - Searching 100 pairs: ['LUNA/USDT', 'ADA/USDT', 'MATIC/USDT', 'CLH/USDT', 'ONE/USDT', 'ROSE/USDT', 'DOT/USDT', 'XRP/USDT', 'SOL/USDT', 'VRA/USDT', 'LINK/USDT', 'KDA/USDT', 'GARI/USDT', 'LTC/USDT', 'DOGE/USDT', 'KOK/USDT', 'MANA/USDT', 'GLMR/USDT', 'FLUX/USDT', 'PBX/USDT', 'STX/USDT', 'QNT/USDT', 'CRO/USDT', 'ICP/USDT', 'CRV/USDT', 'RNDR/USDT', 'TRX/USDT', 'ALGO/USDT', 'QRDO/USDT', 'XLM/USDT', 'BLOK/USDT', 'STRONG/USDT', 'VET/USDT', 'WOO/USDT', 'VR/USDT', 'MOVR/USDT', 'THETA/USDT', 'EOS/USDT', 'EGLD/USDT', 'RUNE/USDT', 'ETC/USDT', 'CHR/USDT', 'DYDX/USDT', 'XMR/USDT', 'IOTX/USDT', 'ENJ/USDT', 'RMRK/USDT', 'ELA/USDT', 'CAKE/USDT', 'ROUTE/USDT', 'TEL/USDT', 'NEO/USDT', 'IMX/USDT', 'PRE/USDT', 'REQ/USDT', 'GALAX/USDT', 'GODS/USDT', 'ELON/USDT', 'BCH/USDT', 'UFO/USDT', 'FIL/USDT', 'UOS/USDT', 'XPR/USDT', '1EARTH/USDT', 'XTZ/USDT', 'NIF/USDT', 'SFUND/USDT', 'OOE/USDT', 'AXS/USDT', 'TRIAS/USDT', 'FTT/USDT', 'XDC/USDT', 'GRT/USDT', 'ENS/USDT', 'SHILL/USDT', 'SOS/USDT', 'KSM/USDT', 'SKL/USDT', 'ZEC/USDT', '1INCH/USDT', 'WILD/USDT', 'ILA/USDT', 'MHC/USDT', 'CIRUS/USDT', 'CHMB/USDT', 'XCAD/USDT', 'ZIL/USDT', 'MIR/USDT', 'CQT/USDT', 'TIME/USDT', 'HERO/USDT', 'XHV/USDT', 'SHA/USDT', 'SENSO/USDT', 'VLX/USDT', 'DAG/USDT', 'DFI/USDT', 'DAO/USDT', 'FLOW/USDT', 'VXV/USDT']
2022-01-18 21:46:02 - AgeFilter - INFO - Removed LUNA/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ADA/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed MATIC/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed CLH/USDT from whitelist, because age 11 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ONE/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ROSE/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed DOT/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed XRP/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed SOL/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed VRA/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed LINK/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed KDA/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed GARI/USDT from whitelist, because age 0 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed LTC/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed DOGE/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed KOK/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed MANA/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed GLMR/USDT from whitelist, because age 7 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed FLUX/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed PBX/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed STX/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed QNT/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed CRO/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ICP/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed CRV/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed RNDR/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed TRX/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ALGO/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed QRDO/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed XLM/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed BLOK/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed STRONG/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed VET/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed WOO/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed VR/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed MOVR/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed THETA/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed EOS/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed EGLD/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed RUNE/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ETC/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed CHR/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed DYDX/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed XMR/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed IOTX/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ENJ/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed RMRK/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ELA/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed CAKE/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ROUTE/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed TEL/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed NEO/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed IMX/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed PRE/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed REQ/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed GALAX/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed GODS/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed BCH/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed FIL/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed UOS/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed XPR/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed 1EARTH/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed XTZ/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed NIF/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed SFUND/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed OOE/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed AXS/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed TRIAS/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed FTT/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed XDC/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed GRT/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ENS/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed SHILL/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed SOS/USDT from whitelist, because age 24 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed SKL/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ZEC/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed 1INCH/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed WILD/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ILA/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed MHC/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed CIRUS/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed CHMB/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed XCAD/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed ZIL/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed CQT/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed TIME/USDT from whitelist, because age 21 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed HERO/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed XHV/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed SHA/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed SENSO/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed VLX/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed DAG/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed DFI/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed DAO/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Removed VXV/USDT from whitelist, because age 4 is less than 31 days
2022-01-18 21:46:02 - AgeFilter - INFO - Validated 5 pairs.

In any case, out of the expected 100 * / USDT pairs, 95 pairs were dropped, which is obvious nonsense, if only because all these 95 pairs were 4 days old.

The glitch mentioned reappears very irregularly, for a while everything seems to be working fine, then there is a case of clearing the whitelist to zero, or a fraction of what it should contain.

After launching my FT for Kucoin without a proxy, it works rather poorly / sluggishly for me, but at least I have an expected list. After starting FT without a proxy, I have in my log:

...
2022-01-18 23:55:44 - AgeFilter - INFO - Removed CLH/USDT from whitelist, because age 11 is less than 31 days
2022-01-18 23:55:44 - AgeFilter - INFO - Removed GARI/USDT from whitelist, because age 0 is less than 31 days
2022-01-18 23:55:44 - AgeFilter - INFO - Removed GLMR/USDT from whitelist, because age 7 is less than 31 days
2022-01-18 23:55:44 - AgeFilter - INFO - Removed SOS/USDT from whitelist, because age 24 is less than 31 days
2022-01-18 23:55:44 - AgeFilter - INFO - Removed TIME/USDT from whitelist, because age 21 is less than 31 days
2022-01-18 23:55:44 - AgeFilter - INFO - Validated 95 pairs.
...

It turns out that pairs that were previously supposed to be 4 days old are now "correct", validated

I've seen signals with similar behavior using your proxy for a long time. I would be very glad if you could look at this problem again ...

Best wishes :-)

P.S. I build the proxy myself on my VPS with

ubuntu@jerzya1ampere-2cpus-2live-usdt:~/kucoin-proxy$ go version
go version go1.17.6 linux/arm64

from version:

commit 3f96b22542fd9dcb8656f4d1717f7e6e34d015be (HEAD -> main, tag: v1.2.6, origin/main, origin/HEAD)
Author: Mikalai Konan <mikalai.konan@icloud.com>
Date:   Fri Jan 14 15:23:58 2022 +0300
mikekonan commented 2 years ago

Thanks for the report.

It will be really good if you will provide the results of the GET klines request. You can find the path in proxy logs enabling verbose 1.

It will be like /kucoin/api/v1/market/candles?symbol=LPT-USDT&type=5min&startAt=1642382225&endAt=1642532225 so you need to curl to proxy, and kucoin API to compare the results.

jurekl commented 2 years ago

Hey Mike, I'll try, of course, but a bit later. I supplemented the info with the proxy and golang versions, because I am trying to build the proxy myself, on my VPS.

As it happens in life, it can be hellishly difficult to find a possible error / defect, because I have been using your proxy from the beginning and only yesterday I discovered such a glaring case as the rejection of 95% coins due to their age. The age of 4 days for 95 coins seems rather unlikely. When I get the data you mention, I will paste it right here!

jurekl commented 2 years ago

I ran two commands:

curl -o response-kucoin.txt 'https://openapi-v2.kucoin.com/api/v1/market/candles?symbol=LPT-USDT&type=5min&startAt=1642382225&endAt=1642532225'

and the next one

curl -o response-proxy.txt 'https://localhost:9000/api/v1/market/candles?symbol=LPT-USDT&type=5min&startAt=1642382225&endAt=1642532225'

Comparing the contents of the two files containing the klines showed that they are identical.

This test of mine is probably of little value, as it would have to be checked online, on the current data, with more or less up-to-date unix timestamp

I downloaded the release https://github.com/mikekonan/exchange-proxy/releases/download/v1.2.1/exchange-proxy_1.2.1_linux_arm64.tar.gz and installed it instead of my own build. I will test this variant for some time ...

response-proxy.txt

response-kucoin.txt .

mikekonan commented 2 years ago

Q: Comparing the contents of the two files containing the klines showed that they are identical. A: You have to call the request for pair that was filtered by age/volatility filter. And I guess that timeframe will be like 1d, for sure you can match the correct request log time.

mikekonan commented 2 years ago

Try this one - https://github.com/mikekonan/exchange-proxy/releases/tag/untagged-5ed2f32e1b9b19addfab

jurekl commented 2 years ago

Sorry Mike, this link https://github.com/mikekonan/exchange-proxy/releases/tag/untagged-5ed2f32e1b9b19addfab

takes me to 404

mikekonan commented 2 years ago

https://github.com/mikekonan/exchange-proxy/releases/tag/v1.2.6

mikekonan commented 2 years ago

There are no fixes for these issues, but more fresh version

jurekl commented 2 years ago

But this is the version (1.2.6) that I have been using for several days (January 14th, I think).

commit 3f96b22542fd9dcb8656f4d1717f7e6e34d015be (HEAD -> main, tag: v1.2.6, origin/main, origin/HEAD)
Author: Mikalai Konan <mikalai.konan@icloud.com>
Date:   Fri Jan 14 15:23:58 2022 +0300
suikula commented 2 years ago

Hello, Im having same problems with the latest version (v1.2.6).

2022-02-09 07:28:13,811 - AgeFilter - INFO - Removed ADA/USDT from whitelist, because age 5 is less than 14 days 2022-02-09 07:28:13,812 - AgeFilter - INFO - Removed ALGO/USDT from whitelist, because age 5 is less than 14 days 2022-02-09 07:28:13,819 - AgeFilter - INFO - Removed AVAX/USDT from whitelist, because age 5 is less than 14 days 2022-02-09 07:28:13,820 - AgeFilter - INFO - Removed AXS/USDT from whitelist, because age 5 is less than 14 days 2022-02-09 07:28:13,820 - AgeFilter - INFO - Removed BTT/USDT from whitelist, because age 5 is less than 14 days 2022-02-09 07:28:13,820 - AgeFilter - INFO - Removed COTI/USDT from whitelist, because age 5 is less than 14 days 2022-02-09 07:28:13,820 - AgeFilter - INFO - Removed CRV/USDT from whitelist, because age 5 is less than 14 days 2022-02-09 07:28:13,821 - AgeFilter - INFO - Removed DOGE/USDT from whitelist, because age 5 is less than 14 days 2022-02-09 07:28:13,821 - AgeFilter - INFO - Removed DOT/USDT from whitelist, because age 5 is less than 14 days 2022-02-09 07:28:13,821 - AgeFilter - INFO - Removed ENJ/USDT from whitelist, because age 5 is less than 14 days

$ go version go version go1.17.3 linux/arm64

VanessaE commented 1 year ago

I have this problem also. It's been a year since the last comment or commit... is this project still being maintained?

mikekonan commented 1 year ago

Greetings! First and foremost, I must mention that I am unable to devote significant time and attention to maintaining the project. The challenges are the inadequacy and instability of the kucoin WebSocket API and the inefficient performance of the lazy initiation approach with freqtrade. An easy and stupid solution would be to implement a health check that enables docker to restart the proxy as required. On the other hand, the optimal solution would involve the distribution of workers that listen to kucoin WebSocket, albeit at the cost of making deployment hard and challenging for the average user.