search

bmino / binance-triangle-arbitrage

Detect in-market cryptocurrency arbitrage
MIT License
1.08k stars 340 forks source link

Depth (0) ? #68

Closed steveeakin closed 4 years ago

steveeakin commented 5 years ago

I keep getting these warnings in my performance.log:

Bid depth (0) or ask depth (0) too shallow to convert ....

I assume that means a trade was prevented, and that something in my config will fix this.

Any reasoning as to why this is happening?

bmino commented 5 years ago

You'll get that warning every time a calculation cycle is attempted that involves a ticker which has not been stored in the local depth cache. This could happen if the some cpu constraints were maxed out and the depth cache update was missed. Likely restarting the bot will fix that issue. You can also increase the DEPTH.INITIALIZATION_INTERVAL

steveeakin commented 5 years ago

👍 Thanks I'll give it a go.

steveeakin commented 5 years ago

Increasing the DEPTH.INITIALIZATION_INTERVAL to 50 gave me this:

Bid depth (50) or ask depth (50) too shallow to convert 163 VIB to ETH using VIBETH

Increasing the DEPTH.INITIALIZATION_INTERVAL to 100 gave me this:

Bid depth (0) or ask depth (0) too shallow to convert 719 MATIC to BNB using MATICBNB

bmino commented 5 years ago

Okay that first error means that a depth of 50 is not sufficient to convert the quantity required. In this case, a depth of 50 is not sufficient to convert 163 VIB. You would need to increase your depth or decrease your max investment size.

In the second case, it looks like a websocket either didn't connect or there are no bids/asks for MATICBNB. This can happen when cpu resources are constrained or limited. You can increase the initialization interval higher to say 300+ which will slow the startup but will not affect calculation times. As a general rule, slower cpus, and low core counts will need higher values to prevent this issue

steveeakin commented 5 years ago

Ok I'll keep tweaking. As a note, this is on a powerhouse server right now so the slow cpu shouldn't come into play here.

On Mon, Jul 8, 2019, 7:33 PM Brandon notifications@github.com wrote:

Okay that first error means that a depth of 50 is not sufficient to convert the quantity required. In this case, a depth of 50 is not sufficient to convert 163 VIB. You would need to increase your depth or decrease your max investment size.

In the second case, it looks like a websocket either didn't connect or there are no bids/asks for MATICBNB. This can happen when cpu resources are constrained or limited. You can increase the initialization interval higher to say 300+ which will slow the startup but will not affect calculation times. As a general rule, slower cpus, and low core counts will need higher values to prevent this issue

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bmino/binance-triangle-arbitrage/issues/68?email_source=notifications&email_token=ABBPN3TOJYAECRZEOV4NAE3P6PFDPA5CNFSM4H5JAS52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZOUSHI#issuecomment-509430045, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBPN3VQUAGW5KBN55I37JTP6PFDPANCNFSM4H5JAS5Q .

bmino commented 5 years ago

Good to know. If it happens again, check the Binance webclient for the ticker in question and make sure that it wasn't taken offline and has sufficient bids/asks listed

steveeakin commented 5 years ago

Bid depth (0) or ask depth (0) too shallow to convert 228 BCPT to ETH using BCPTETH Bid depth (0) or ask depth (0) too shallow to convert 1.307 ZEN to ETH using ZENETH

These 2 happened twice today, a minute apart (7:46 & 7:47, each)

Looking through the hundreds of logs that look like this since last night, it all appears to be the second leg of a trade matching my config criteria.

I see it happening for WAVESBNB, VIBEBTC, STRATETH, etc. All seem to have sufficient bids / asks

steveeakin commented 5 years ago

Testing on my local machine today, I see the same things in the performance log.

Bid depth (0) or ask depth (0) too shallow to convert 0.042 ETH to BTC using ETHBTC

bmino commented 5 years ago

Hmmm, can I get a copy of your config (without your api/secret)?

steveeakin commented 5 years ago 
{
    "KEYS": {
        "API": "xxxx",
        "SECRET": "xxxx",
    },

    "INVESTMENT": {
        "BASE": "BTC",
        "MIN": 0.001,
        "MAX": 0.02,
        "STEP": 0.002
    },

    "TRADING": {
        "ENABLED": true,
        "EXECUTION_STRATEGY": "linear",
        "EXECUTION_TEMPLATE": ["BUY", "SELL", "SELL"],
        "EXECUTION_CAP": 0,
        "TAKER_FEE": 0.001,
        "PROFIT_THRESHOLD": 0.2,
        "AGE_THRESHOLD": 200,
        "WHITELIST": []
    },

    "HUD": {
        "ENABLED": true,
        "ARB_COUNT": 25
    },

    "LOG": {
        "LEVEL": "warn",
        "PRETTY_PRINT": true
    },

    "DEPTH": {
        "SIZE": 100,
        "PRUNE": false,
        "INITIALIZATION_INTERVAL": 300
    },

    "CALCULATION_COOLDOWN": 100
}
bmino commented 5 years ago

I wonder if turning off the HUD would affect this. I traditionally don't use the HUB except for initial testing for releases.

That uses extra compute cycles after every calculation cycle to sort, filter, and paint the screen with the top arbs

steveeakin commented 5 years ago

HUD is off for the production server (this was a cloned config with HUD = on from my local box.)

Still, I'll run it now on my local box with HUD off to confirm.

steveeakin commented 5 years ago

["2019-7-12 14:03:30"] WARN : Bid depth (0) or ask depth (0) too shallow to convert 0.001 BTC to LTC using LTCBTC

steveeakin commented 5 years ago

I put some debugging in CalculationNode.js inside of orderBookConversion() before the thrown exception.

console.log('amountTo: ' + amountTo);
console.log('amountFrom: ' + amountFrom);
console.log('quantity: ' + quantity);
console.log('rate: ' + rate);
console.log('exchangeableAmount: ' + exchangeableAmount);
console.log('----------');
console.log('Ticker: ' + ticker);
console.log('symbolFrom: ' + symbolFrom);
console.log('symbolTo: ' + symbolTo);
console.log('askRates:');
console.log(askRates);

This outputs:

amountTo: 0
amountFrom: 0.001
quantity: undefined
rate: undefined
exchangeableAmount: undefined
----------
Ticker: LTCBTC
symbolFrom: BTC
symbolTo: LTC
askRates:
[]
bmino commented 5 years ago

Good work. I'll take a look at the specifics when I can set aside some time. Come to mention it, I think something similar to this was mentioned before perhaps in jaggedsoft's api project. This is likely a race condition or bug, but looks legit based upon your findings

steveeakin commented 5 years ago

I may have some time to look deeper in the coming days. Any tips to help reduce my back-tracing these variables?

On Mon, Jul 15, 2019 at 12:31 PM Brandon notifications@github.com wrote:

Good work. I'll take a look at the specifics when I can set aside some time. Come to mention it, I think something similar to this was mentioned before perhaps in jaggedsoft's api project. This is likely a race condition or bug, but looks legit based upon your findings

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bmino/binance-triangle-arbitrage/issues/68?email_source=notifications&email_token=ABBPN3SEI4KQEMG7BPSHD73P7SQ4RA5CNFSM4H5JAS52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ6HTSQ#issuecomment-511474122, or mute the thread https://github.com/notifications/unsubscribe-auth/ABBPN3SWWL3ZOQ64X4HJNJLP7SQ4RANCNFSM4H5JAS5Q .

-- Be awesome,

bmino commented 5 years ago

My advice would be to put conditional breakpoints to determine when the depth of a ticker is depleted. You could also use the HUD to generate depth snapshots of all tickers. A few versions back there was pseudo code checked in to do this

steveeakin commented 5 years ago

Going deeper into debugging:

7/19/2019, 10:16:05 AM  [[min] - [max]] ~ [avg]
7/19/2019, 10:16:05 AM  DEBUG:  Calculation times: [17 - 57] ~ 20 ms
7/19/2019, 10:16:05 AM  DEBUG:  Bid depth cache size: [5 - 20] ~ 19
7/19/2019, 10:15:20 AM  DEBUG:  Ask depth cache size: [6 - 20] ~ 19
7/19/2019, 10:14:36 AM  WARN :  Bid depth (20) or ask depth (19) too shallow to convert 2728 IOTX to ETH using IOTXETH
MatsamaFFCC commented 5 years ago

Hi, Same issue for me. I'm using the script for 3 days but no one trade was done; but per day, I have tons of "Bid depth (0) or ask depth (0) too shallow to convert" messages in the performances log file. Tried to follow the recommandations on this page but actually not changing anything : increasing the initialization interval to 300 (even 500), turning off the HUD, decreasing the number of tickers in the whitelist... I don't know what to do, help :(

My today's config file: ` "INVESTMENT": { "BASE": "BTC", "MIN": 0.001, "MAX": 0.002, "STEP": 0.0005 },

"TRADING": {
    "ENABLED": true,
    "EXECUTION_STRATEGY": "linear",
    "EXECUTION_TEMPLATE": ["BUY", "SELL", "SELL"],
    "EXECUTION_CAP": 0,
    "TAKER_FEE": 0.00,
    "PROFIT_THRESHOLD": 0.30,
    "AGE_THRESHOLD": 300,
    "WHITELIST": ["BTC","ETH","DAI","USDT","TUSD","XMR","BNB","MATIC","ETH","LINK","LTC","DUSK","COS","XRP","EOS","BCHABC","ADA","ONE","BCPT","TRX","REN","ALGO","XMR","BTT","NANO","CELR","RVN","ENJ","BAT","WAVES","KMD","FTM","AGI","DASH","ATOM","ANKR","WTC","ZEC","LUN","ERD","XLM","NEO","VET","ARN","ELF","WIN","ZIL","MDA","IOTA","VIB","THETA","BQX","EVX","ETC","ZRX","GXS","QKC","LRC","POLY","DENT","ICX","ADX","IOTX","PPT","REP","INS","ONG","NXS","ARDR","LOOM","PIVX","APPC","TRIG","NAV","WINGS","EDO","TUSDB","BGBP","DOGE","USDSB","BTCB","PHB","BCPT","TFUEL","FET","USDS","CBM","MITH","USDC","DCR","PAX","GO","HC","ONG","VTHO","DOCK","HOT","IQ","EOP","AE","AION","NAS","WPR","GNT","BRD","WAN"]
},

"HUD": {
    "ENABLED": true,
    "ARB_COUNT": 5
},

"LOG": {
    "LEVEL": "info",
    "PRETTY_PRINT": true
},

"DEPTH": {
    "SIZE": 50,
    "PRUNE": false,
    "INITIALIZATION_INTERVAL": 300
},

"CALCULATION_COOLDOWN": 250

}`

bmino commented 4 years ago

Might have finally gotten towards the bottom of this. Stay tuned. I'm investigating some potential fixes!

bmino commented 4 years ago

Here is what happens when a websocket disconnects:

  1. Current depth cache is deleted
  2. New websocket subscription completes
  3. Depth request is made to initialize new depth cache
  4. Depth request completes and depth cache is initialized with snapshot
  5. Websocket data continues to update depth cache

This process could take up to a few thousand milliseconds depending on your internet connection speed and latency to Binance. Between steps 1 and 4, any and all calculation attempts that involve the ticker that was disconnected will throw depth(0) errors appropriately because there is no depth data for that ticker.

The bulk of the wait time is until the https://api.binance.com/api/v1/depth api REST call completes. I've been parsing a few logs where I have forced this scenario and confirmed that depth(0) errors only occur during the same time window after a websocket disconnects and reconnects.

kutlay commented 3 years ago

I'm still having this error. Here is my config:

"INVESTMENT": { "USDT": { "MIN": 10, "MAX": 20, "STEP": 5 } },

"SCANNING": {
    "DEPTH": 10,
    "WHITELIST": []
},

"EXECUTION": {
    "ENABLED": true,
    "CAP": 1,
    "STRATEGY": "linear",
    "TEMPLATE": ["BUY", "SELL", "SELL"],
    "FEE": 0.075,
    "THRESHOLD": {
        "PROFIT": 0.00,
        "AGE": 25
    }
},

"HUD": {
    "ENABLED": true,
    "ROWS": 10,
    "REFRESH_RATE": 500
},

"LOG": {
    "LEVEL": "debug",
    "PRETTY_PRINT": true,
    "STATUS_UPDATE_INTERVAL": 2
},

"WEBSOCKET": {
    "BUNDLE_SIZE": 1,
    "INITIALIZATION_INTERVAL": 1000
},
bmino commented 3 years ago

@kutlay please open a new issue. I think you have a different cause but we can chat there