File "jesse_live\exchanges\LiveExchange.py", line 367, in jesse_live.exchanges.LiveExchange.LiveExchange._connect_ws._on_message

[qwpto]

Describe the bug

ERROR:
Traceback (most recent call last):
  File "jesse_live\exchanges\LiveExchange.py", line 367, in jesse_live.exchanges.LiveExchange.LiveExchange._connect_ws._on_message
  File "jesse_live\exchanges\Binance\BinancePerpetualFuturesMain.py", line 380, in jesse_live.exchanges.Binance.BinancePerpetualFuturesMain.BinancePerpetualFuturesMain.on_ws_message
  File "jesse_live\exchanges\Binance\BinancePerpetualFuturesMain.py", line 482, in jesse_live.exchanges.Binance.BinancePerpetualFuturesMain.BinancePerpetualFuturesMain.prepare_candle_stream
  File "jesse_live\exchanges\LiveExchange.py", line 1080, in jesse_live.exchanges.LiveExchange.LiveExchange.add_candle
  File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python39\lib\site-packages\jesse\store\state_candles.py", line 187, in add_candle
    f"Could not find the candle with timestamp {jh.timestamp_to_time(candle[0])} in the storage. Last candle's timestamp: {jh.timestamp_to_time(arr[-1])}. timeframe: {timeframe}, exchange: {exchange}, symbol: {symbol}"
  File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python39\lib\site-packages\jesse\helpers.py", line 776, in timestamp_to_time
    return str(arrow.get(timestamp / 1000))
  File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python39\lib\site-packages\arrow\api.py", line 91, in get
    return _factory.get(*args, **kwargs)
  File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python39\lib\site-packages\arrow\factory.py", line 267, in get
    raise TypeError(f"Cannot parse single argument of type {type(arg)!r}.")
TypeError: Cannot parse single argument of type <class 'numpy.ndarray'>.

To Reproduce live trading on Binance Perpetual Futures 2022/09/16 18:52 that led to the following message simultaneously in multiple instances (BTC-USDT, and ETH-USDT) being raised to the discord channels at 30x per minute each, until I got home to stop it around 21:00 UTC

Expected behavior Single error message, then recovery preferably without interruption otherwise restarting the strategy, or perhaps stopping.

Screenshots N/A

Enviroment (please complete the following information):

• OS: Windows 11
• Version jesse, version 0.39.4 and jesse_live v0.39.1

Additional context System had been running for 3 days on a 1min timeframe. Previously, same setup had been running for 2 weeks.

[qwpto]

I think the discord posts were rate limited at some point, in case it occurs repeatedly it's possible to get banned at the CloudFlare level. https://discord.com/developers/docs/topics/rate-limits So it would be good to also rate limit and perhaps use the retry after response to slow things down in cases like this.

[saleh-mir]

I appriciate the report but in case you got this error on the GUI dashboard, it would be way more helpful if you reported it via the "report" button. That way I get to see useful debuging detail such as installed versions, the full Jesse logs, plus raw exchange logs. Never the strategy of course.

I need to know the routes in this session, whether or not the "local candle generation" was enabled.

[qwpto]

b2a2ac4a-20a5-46f3-8c4e-083bcb798634.txt

[qwpto]

Yeah I just saw from the other feedback that this button existed, however I had shut them down immediately when I got home. Hopefully the log above contains the info needed.

You can see in there it did indeed exceed the discord rate limit:

[INFO | 2022-09-16T18:56:45] Discord ERROR [429]: {
  "global": false, 
  "message": "The resource is being rate limited.", 
  "retry_after": 3208
}

[saleh-mir]

Yeah I just saw from the other feedback that this button existed, however I had shut them down immediately when I got home. Hopefully the log above contains the info needed.

You can see in there it did indeed exceed the discord rate limit:

[INFO | 2022-09-16T18:56:45] Discord ERROR [429]: {
  "global": false, 
  "message": "The resource is being rate limited.", 
  "retry_after": 3208
}

I suggest using Telegram instead which has higher limits.

[qwpto]

One every 2 seconds is probably ok though, at that point it really is spamming.

[qwpto]

You can see at the end of the log I need to account for the ban in the JesseReportDiscord addon, since this crashes at the moment under this scenario. I'll make that update.

[qwpto]

Ok, that's now fixed in version 1.2.1 of the JesseReportDiscord package. Also added timouts so it doesn't wait forever.

[qwpto]

I have 64GB of DDR5 memory, but there was a webpage that had run out of memory. When I saw the stats though the load was low (less than 50% consumption). It's happened again today interestingly. Same error. Ran fine at 1min for 5 hours though.

[qwpto]

Actually that's interesting, it recovered in a way:

[INFO | 2022-09-17T16:33:00] Executing ... then this error for 17 mins, then

[INFO | 2022-09-17T16:50:00] Executing ... [INFO | 2022-09-17T16:50:22] Error fetching candles via REST for ATOM-USDT from Binance Perpetual Futures. No worries, will try later. Error: "HTTPSConnectionPool(host='fapi.binance.com', port=443): Max retries exceeded with url: /fapi/v1/klines?interval=1m&symbol=ATOMUSDT&startTime=1663432741000&endTime=1663492681000&limit=1000 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000001894F784D60>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))" [INFO | 2022-09-17T16:50:50] Error fetching candles via REST for ATOM-USDT from Binance Perpetual Futures. No worries, will try later. Error: "HTTPSConnectionPool(host='fapi.binance.com', port=443): Max retries exceeded with url: /fapi/v1/klines?interval=1m&symbol=ATOMUSDT&startTime=1663432769000&endTime=1663492709000&limit=1000 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000001894F7876A0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))" [INFO | 2022-09-17T16:51:00] Executing ... [INFO | 2022-09-17T16:51:18] Error fetching candles via REST for ATOM-USDT from Binance Perpetual Futures. No worries, will try later. Error: "HTTPSConnectionPool(host='fapi.binance.com', port=443): Max retries exceeded with url: /fapi/v1/klines?interval=1m&symbol=ATOMUSDT&startTime=1663432797000&endTime=1663492737000&limit=1000 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000001894F787BE0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))" [INFO | 2022-09-17T16:51:46] Error fetching candles via REST for ATOM-USDT from Binance Perpetual Futures. No worries, will try later. Error: "HTTPSConnectionPool(host='fapi.binance.com', port=443): Max retries exceeded with url: /fapi/v1/klines?interval=1m&symbol=ATOMUSDT&startTime=1663432825000&endTime=1663492765000&limit=1000 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000001894F784130>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))" [INFO | 2022-09-17T16:52:00] Executing ...

which continued like that with it working and then not for a bit until [INFO | 2022-09-17T17:03:58] Then it went into this reported error again

and eventually back into this oscillation of errors fetching candles and running.

[qwpto]

Ok, have just seen the documentation update: https://jesse.trade/help/faq/can-i-run-multiple-instances-of-jesse-at-the-same-time My instances here were running native and sharing the same redis and postgres databases, using different coins, but perhaps that has caused an issue.

[saleh-mir]

I looked at the logs again. It's like all the candle fetching requests are failing and the store stays empty. Seems like a connection issue.

Also it's weird that the log value related to "local candle generation" is absent in your logs. Did you remove it? Is the debug mode enabled (the value for that is absent as well).

1. Did it occur again?
2. Did you try it after isolating database and redis?

[qwpto]

Thanks @saleh-mir, It should have been with this config:

1. Yes, within a few hours the second time. Still using the same DBs (before I found out)
2. Still just running one agent, with a new set of separate databases. I'll try multiple again soon, just wanted to see that I was back to stable before trying as I changed which redis and postgres I was using to make it easier to start multiple instances. I'll post here again on how it goes once I do, good or bad.

[saleh-mir]

Thank you. I will close this for now then.

[qwpto]

Has been running with with multiple instances using a redis and postsgres for each for 2 days, the candle error with the exchange occured and did not cause a problem like before. So it seems this was the issue.