Bot is getting slow or hags out

[peweb]

Version

v0.0.96 (209d2e0)

Description

Bot is running on a VPS with very low load. If i dont touch it for several days I cant login to the frontend or if I login it is slow. If I reboot the server everything works fine.

How can I troubleshoot and fix this?

I have 0 /usr/bin/docker restart binance-bot tradingview

[habibalkhabbaz]

Hello @peweb

Yes it's valid issue. It is happening continuously after migrating to the custom queue implementation. I will look into this when I have time or maybe @uhliksk can help us on this too.

[uhliksk]

Hello @habibalkhabbaz, @peweb

Do you have any logs? I don't have this issue. Maybe unstable internet connection can cause this? I'll try to do some tests.

[uhliksk]

I probably found the cause of this issue. Cronjob is not deterministic as it can interrupt the job if it's running for more than 20 seconds which will cause the job will not indicate it's finished. I'll rework the code to finish the job properly if running for more than 20 seconds.

https://github.com/chrisleekr/binance-trading-bot/blob/d517846d7671b230b0bfaaa4f7a42b7b37804aaa/app/server-cronjob.js#L62-L69

[shhhmel]

Same problem. docker restart binance-bot tradingview and cronjob doesn't help. UI still loading infinitely.

[shhhmel]

Possible solution:

Instead of using setTimeout to interrupt the job if it's running for more than 20 seconds, you could use the async-timeout library to handle the timeout.

const fulfillWithTimeLimit = async (logger, timeLimit, task, failureValue) => {
  let timeout;
  try {
    timeout = asyncTimeout.set(timeLimit, task);
    return await timeout;
  } catch (err) {
    logger.error(
      { tag: 'job-timeout' },
      `Failed to run the job within ${timeLimit}ms.`
    );
    return failureValue;
  } finally {
    asyncTimeout.clear(timeout);
  }
};

This way, if the task takes longer than 20 seconds to complete, it will throw a TimeoutError which will be caught by the catch block. The job will then proceed with the failureValue.

[peweb]

@chrisleekr Can you please check this and release it if this is fixed? Thanks!

[chrisleekr]

@peweb, The proposed code by @shhhmel is doing the same task as what the current code does.

[uhliksk]

@chrisleekr I'm sorry I forgot to push the PR for this. I little bit busy last few weeks so I'll push what I tried to do, but I didn't have a time to finish it yet.

[dasbts]

If this works, then you are a beast, because I get this issue every day (sometimes multiple times a day) even with cronjob to restart the bot every 6h. Either it freezes with the cache locks, or it just stops working, alternatively random symbols websockets stop working.

I am currently trying this out and will update how it looks in the next days. However, I had to change in the queue.js as I got: setTimeout(...).then is not a function and since it is just a timeout, I changed it From:

        setTimeout(r, 20000).then(
          logger.error({ symbol }, `Queue ${symbol} job timeout`)
        )

Into:

      new Promise((resolve) => {
        setTimeout(() => {
          resolve();
        }, 20000);
      }).then(() => {
        logger.error({ symbol }, `Queue ${symbol} job timeout`);
      })

I ran it for only like 30 minutes and got error "javascript heap out of memory docker mark-sweep reduce last resort gc in old space" even tho I have a lot of RAM available and also ZSwap which compresses my RAM. However, it was apparently as easy to fix on Linux as writing in terminal: export NODE_OPTIONS=--max_old_space_size=4096 On windows, save it as an environment variable.

EDIT: Sadly didn't resolve the random "action" timer stopping (thus meaning the ws stopped/restarted) for random symbol at random time.

[dasbts]

Applied the code in your commit @uhliksk, but today I noticed it is still doing this. It is very slow and having issues releasing the locks (which I have retries for to not totally break the bot when it cannot release redlock), so when changing pages for the symbols it even went back to the initial page for some seconds, before going to the page I chose.

Many of my symbols also stopped the tickers at random times 2-3 hours ago.

Nice try tho!

[uhliksk]

@dasbts Thank you for testing. I'm not able to finish the thing because of some health issue. I'll be glad for any suggestions until I'll be able to get back to work again.

[dasbts]

@uhliksk I had to revert the change sadly, for some reason when I was selling a trade, it thought the last buy price was 0 even tho it was recorded all this time, so the profit was reported as way more than it was worth. Also had a trade trigger at the sell, then for some reason it deleted the trade and everything related to it, but didn't sell. Of course only had to remove the sell history and add back the last buy price, but sadly missed a profitable sell opportunity.

Not 100% sure if this was actually related, but I started some other strange errors too. Haven't really had any problems like this after I reverted and after implementing the https://github.com/chrisleekr/binance-trading-bot/issues/629 changes (which I just need to know how you want PR's made as some repos seem to have develop branches etc.

However, my changes included removing the WS tickers subscription for all BTC equivalents (if trading for example DOGE/USDT), as used for dust transactions, as otherwise, everytime I would restart a socket, it would also add i.e. DOGE/BTC to my GUI cards, even tho it's not monitored in the config. After doing this I also got a big performance boost in that it previously hanged up and was slow in general. So for me that's a clue something is suspect with the BTC dust part in tickers.js.