Bot will stop updating after the internet connection is interrupted

[uhliksk]

Version

v0.0.89(165d110)

Description

After the internet connection is interrupted the bot will stop updating and it is not able to reconnect again.

To Reproduce

1. Start the bot
2. Disconnect the router from Internet
3. After a while connect the router back
4. There is no update since then

Expected Behaviours

If there is no update for specified time the bot should try to reconnect or restart itself.

Screenshots

Bot stopped updating 3 hours ago when internet connection outage occured because of local storm.

Additional context

Sometimes after an outage symbols are not updating but TradingView is updating fine. Sometimes after an outage TV is not updating but symbols are updating fine. Sometimes nothing is updated after an outage at all.

[habibalkhabbaz]

Hello @uhliksk

Do you see any error log?

Regarding the Binance websocket implementation, it has a reconnect mechanism and it should reconnect by default

https://github.com/ViewBlock/binance-api-node/pull/49

If you have any log will be helpful.

[uhliksk]

Hello @habibalkhabbaz

There is no error logged. I found it's enough to break the connection even for a few seconds and it is not able to recover. I have to restart docker container to get back the connection.

Edit: When I tried to disable trading and then enable again, this single symbol will start updating again.

[habibalkhabbaz]

@uhliksk

I will try to disconnect when I have time and I will report back to you. Maybe we have to apply a fix for this.

[chrisleekr]

Oh hmm, I had a code to restart automatically if does not receive candle info for a certain period. We may need to reintroduce that. Let me dig into how I did it previously.

@habibalkhabbaz

Here, https://github.com/chrisleekr/binance-trading-bot/blob/v0.0.85/app/server-binance.js#L109

So what it did is, if didn't receive the candle for a min, then it was resetting the WebSocket. Maybe we can reapply this.

[Rayn0r]

I experienced the same problem with the python implementation of the websocket. As long as you keep your IP and the interruption is only short, chances are high, the socket will reconnect. Upon disconnect, you need to call client.close_connection() and execute your main() or whatever again. Since this will be triggered while your connection is down, you will also have to catch any exceptions while trying to reconnect... Here's how I did it in python, if it is any help:

async def handle_socket_message(msg, client):
    """
    function, that is called as soon as a new candles comes in
    """
    if msg['e'] == 'error':
        await disconnect_callback(client=client)
        print('Error occurred when processing message.\n')
    else: ...

async def main():
    """
    main loop for receiving candle data and processing them
    """
    while True:
        try:
            client = await AsyncClient.create()
        except(aiohttp.client_exceptions.ClientConnectorError, socket.gaierror, asyncio.TimeoutError) as error_msg_a:
            print(f"error: {error_msg_a}")
            # should we add a sleep here?
            time.sleep(3)
        else:
            print("finally got through the loop")
            break

    try:
        bi_socket_manager = BinanceSocketManager(client, user_timeout=60)
    except(aiohttp.client_exceptions.ClientConnectorError, socket.gaierror) as error_msg_0:
        print(f"Error! in main loop 0:\n{error_msg_0}")
        await disconnect_callback(client=client)

    # start any sockets here, i.e a trade socket
    kline_candles = bi_socket_manager.kline_socket(gv.SYMBOL, interval=client.KLINE_INTERVAL_15MINUTE)

    # start receiving messages
    try:
        status = await client.get_system_status()
        print(f"status: {status['msg']}\n")

        async with kline_candles as stream:
            while True:
                try:
                    res = await asyncio.wait_for(stream.recv(), timeout=60)  # create/await response
                    await handle_socket_message(msg=res, client=client)  # process message
                except (asyncio.TimeoutError, websockets.exceptions.ConnectionClosed, asyncio.CancelledError, asyncio.exceptions.CancelledError, asyncio.exceptions.TimeoutError) as error_msg_1:
                    print(f"Error! in main loop 1:\n{error_msg_1}")
                    await disconnect_callback(client=client)

    except BinanceAPIException as error_msg_2:
        print(f"Error! in main loop 2:\n{error_msg_2}")
        await disconnect_callback(client=client)

async def disconnect_callback(client):
    """
    method to call for a reconnect
    """
    print("Error! disconnect_callback() got called")
    await client.close_connection()  # close connection
    time.sleep(60)  # wait a minute before restarting
    await main()  # restart client and kline socket

[Rayn0r]

Dunno how to do this in node.js, but in python this line has a built in timeout: res = await asyncio.wait_for(stream.recv(), timeout=60) # create/await response It triggers a connection timeout exception which is handled below and is more convenient than to check for incoming data over a certain period of time.

[uhliksk]

So what it did is, if didn't receive the candle for a min, then it was resetting the WebSocket. Maybe we can reapply this.

@chrisleekr I think this is not very good solution as a lot of symbols are not traded for more than a minute. Better solution will be to trigger the websocket reset only when all symbols didn't received an update for a few minutes. If there is no update for all symbols then it is more likely connection error.

[habibalkhabbaz]

Hello guys @uhliksk @chrisleekr @Rayn0r Sorry for the late reply. As mentioned by @chrisleekr, that would solve the problem. It was removed in the latest releases due to massive refactoring.

@chrisleekr I think this is not very good solution as a lot of symbols are not traded for more than a minute

@uhliksk I think it will work as you expect.

I will implement that when I have time.

[dasbts]

Worked on this for way too much, see most recent responses in: https://github.com/chrisleekr/binance-trading-bot/issues/619