Issue with liquidations

[krazzarkov]

Hello,

I've been playing around and testing out the liquidator bot. So far it's pretty awesome and I've enjoyed learning about how it works and the general liquidation process. That being said, I haven't actually been successful in performing a liquidation. I have run the bot for 24+ hours and have come across quite a few errors which I wanted to point out below.

The most common error I've encountered is Transaction failed: MangoErrorCode::InvalidParam. From my understanding, this error results when you were not the first to execute the liquidation as mentioned in Issue #7

Fri, 03 Dec 2021 19:18:18 GMT  sending tx  4vdD9PBUX86FMCScc55gNLPnym1Wpi1Pk3GMC7He8mAsDtmgEcTqkbj1hjY9G2BAdMr9u8yx5r3UcDBt2ct5VKSG
REST error for 4vdD9PBUX86FMCScc55gNLPnym1Wpi1Pk3GMC7He8mAsDtmgEcTqkbj1hjY9G2BAdMr9u8yx5r3UcDBt2ct5VKSG {
  confirmationStatus: 'processed',
  confirmations: 0,
  err: { InstructionError: [ 0, [Object] ] },
  slot: 109910945,
  status: { Err: { InstructionError: [Array] } }
}
Failed to execute trigger order for 31mrAbssy2w5ZpDgJgQU7UAFfRPSTbQNBXrTqn1LUyEM Error: Transaction failed: MangoErrorCode::InvalidParam; src/processor.rs:4524
    at MangoClient.<anonymous> (C:\Users\zarko\Desktop\MangoLiq\liquidator-v3\node_modules\@blockworks-foundation\mango-client\src\client.ts:276:21)
    at Generator.next (<anonymous>)
    at fulfilled (C:\Users\zarko\Desktop\MangoLiq\liquidator-v3\node_modules\@blockworks-foundation\mango-client\lib\src\client.js:5:58)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)

Besides this error, I have experienced the following timeout issue. This error usually results after the CLI spams sending tx for a few minutes.

Timed out for txid 3W1yF5z5Hn64DuxC7vyibXxbuEysvBk7XjmQNChtczLT5ZFsBARzZW9Rj3oA75oVeexw3po2GfPQsXHYZaB4yMpV
Failed to execute trigger order for 3mZEubS5b3wyZrVSrALU1JpEUdnFELAvFzdyEmCYYbDT Error: Timed out awaiting confirmation on transaction

I've also experienced this error as well: Failed to liquidate account 3cj8aQtAFe2rzXhYVP2wUBZJVutSeKJDCh3bkjwJukce Error: "BlockhashNotFound"

This one: Could not match OpenOrdersAccount to MangoAccount

And this one as well: Failed to liquidate account 8zq5hsJgtCDvMe432N1gzjmyetzrutse8XwjgeWCyFCh Error: Transaction failed: MangoErrorCode::InvalidPriceCache Cache the oracle price to resolve; src/state.rs:588

I'm not sure if all of these errors are related or are completely different issues.

Finally, after observing the logs and going through them, I thought I had found a few examples of where the bot managed to liquidate successfully as the following was displayed on the logs:

Watching accounts... Liquidated account 7nx81DsbC7rmANwZbykVxR3TsQ1HXBhVhAaGYi5TKt41

Unfortunately, after checking Mango Markets > History > Liquidations nothing shows up and the starting balance has not changed.

I apologize for the long post. I was just curious if you could provide some more insight on why these errors are occurring and also if this is normal or out of the ordinary. The bot has been running during a pretty volatile day in the markets so I excepted to have at least one successful liquidation.

Any help would be greatly appreciated. Thank you in advance!

EDIT: I'm using the default .env config. I have also tried playing around and modifying INTERVAL, INTERVAL_ACCOUNTS, and INTERVAL_WEBSOCKET so they are faster however, the same issues still persist. My test mango account currently has a small amount of $140 USDC as collateral.

[krazzarkov]

UPDATE: I was running the bot during the huge flash crash yesterday and the bot managed to pick up a few liquidations! It was super cool to see it working but it seems way harder to get a successful liquidation during normal/less volatile periods.

I'm guessing this has to do with the fact that there are other people doing the same thing with more advanced setups and private RPCs? Curious to see how the bot can be improved so that it is competitive/functional in all market conditions besides huge liquidation events.

I'm also still trying to understand the code and logic but during the time I was running the bot, I also noticed another piece of weird behavior I still can't understand.

For the past 10-12 hours the bot has been spamming out transactions constantly for the same two sick accounts which are not in a liquidatable state. It goes through a loop of sending out a transaction, having it fail since the accounts are not liquidatable, and then retrying the same thing. Perhaps the two accounts are glitched on Mango's end and it's messing with the bot?

Screenshots below for the two accounts below:

Thank you again!

[riordanp]

Hi @krazzarkov

The most common error I've encountered is Transaction failed: MangoErrorCode::InvalidParam

You're correct, this one is someone getting there before you when attempting to trigger an advanced order. Will be removing this error message in a future version as it's not particularly helpful.

Besides this error, I have experienced the following timeout issue. This error usually results after the CLI spams sending tx for a few minutes.

Failed to liquidate account 3cj8aQtAFe2rzXhYVP2wUBZJVutSeKJDCh3bkjwJukce Error: "BlockhashNotFound"

These are both due to RPC node problems. This is quite common especially with public RPC services.

Watching accounts... Liquidated account 7nx81DsbC7rmANwZbykVxR3TsQ1HXBhVhAaGYi5TKt41

Not totally sure on this one, adding some new logging soon that should make it clearer what happens in this case.

Failed to liquidate account 8zq5hsJgtCDvMe432N1gzjmyetzrutse8XwjgeWCyFCh Error: Transaction failed: MangoErrorCode::InvalidPriceCache Cache the oracle price to resolve; src/state.rs:588

This is a wider issue with Mango at the moment especially in periods of high load. You would have liquidated but the price cache was stale.

I have also tried playing around and modifying INTERVAL, INTERVAL_ACCOUNTS, and INTERVAL_WEBSOCKET so they are faster however

Changing INTERVAL_ACCOUNTS and INTERVAL_WEBSOCKET will not really make any difference to you liquidation success nor will having a higher account value.. They just control how long to wait between reloading the account list (most account updates come in on websocket anyway, this is just a failsafe), and how long to wait before reconnecting to the websocket. The loop that reloads prices and checks accounts health will run at the same speed regardless.

I was running the bot during the huge flash crash yesterday and the bot managed to pick up a few liquidations! It was super cool to see it working but it seems way harder to get a successful liquidation during normal/less volatile periods.

Glad you got some liquidations! Most of the time when the market is more stable there are very few liquidations happening at all. During periods of volatility there is a large amount. This has been my experience too. There are no doubt people out there running private RPCs with very low latency to the liquidator, and probably some with their own liquidator implementations or modifications to make it faster.

For the past 10-12 hours the bot has been spamming out transactions constantly for the same two sick accounts which are not in a liquidatable state.

Might be caused by RPC caching. Which endpoint do you use? Can you try a different one?

[krazzarkov]

Hi @riordanp,

Thank you for getting back to me and answering all my questions!

For the error messages, all your explanations made sense to me.

Regarding my last issue with the same two accounts trying to be liquidated by the bot, it definitely seems to be an RPC caching issue. I tried testing and switching from https://solana-api.projectserum.com which is the RPC endpoint I have been using, to a free node from QuickNode. This fixed the issue.

The problem is that this bot creates a lot of calls and is not able to be used with most free RPC's at all or for more than a day due to rates or amount of calls per month being limited.

To my knowledge, the only free public endpoint that isn't rate-limited is https://solana-api.projectserum.com. Is there a way to fix the RPC caching issue directly for the serum endpoint?

Thank you!

[riordanp]

If you simply remove the ENDPOINT_URL environment variable it will default to using the Mango node, though this can get overloaded at times too. Shouldn't get caching problems there though.

Many liquidator operators are paying for private RPC for greater reliability.

[krazzarkov]

@riordanp Yup, removing the ENDPOINT_URL seems to have fixed the problem. For some reason, I thought that the default RPC was the Serum one.

I'll look into some private RPC solutions. They seem to be quite expensive but, I'm curious to try it out. Maybe I can deploy both the Liquidator bot + RPC on Google Cloud and see how that goes.

Thank you again for the help and the awesome bot! :)