Number of WebSockets seems high [relates to issue #98]

[dlasher]

( see https://github.com/bmino/binance-triangle-arbitrage/issues/98 )

if we find XX triangle relationships, why does it open max amount of websockets? Shouldn't the math be something like WebsocketCount = (Triangle Relationships * 2) +1

Found 625/882 currently trading tickers Found 158 triangular relationships Checking configuration ... Checking balances ... Opening 625 depth websockets ...

in your example 158 triangular relationships are found, but let's say that only 1 relationship was found. We can call that relationship A -> B -> C. This requires AB, BC, and CA which would require three websockets to track those order books. This is why you see more web socket connections than triangular relationships

Trying this out visually:

ETH-TRX-BTC-ETH - Track ETH-TRX, TRX-BTC and BTC-ETH ETH-ADA-BTC-ETH - Track ETH-ADA, ADA-BTC and BTC-ETH ETH-LINK-BTC-ETH - Track ETH-LINK, LINK-BTC and BTC-ETH so for those (3) triangle relationships, I see 7 separate websockets.

In other words CA gets reused, and you're tracking lots of unique AB and BC? (2 more triangle, + 1 shared?)

Even dropping the whitelist down to 50, produced a lot:

Found 641/906 currently trading tickers Found 50 triangular relationships Checking configuration ... Checking balances ... Opening 189 depth websockets ...

Assuming ZERO overlap, that's still 3.78x the number of relationships. In practice there should be a lot of overlap. Shouldn't it be around 101 Websockets for 50 relationships?

[bmino]

Upon some investigation I found a few things:

1. We are misusing the js array type in one instance (smh haha)
2. When not whitelisting, websocket connections are opened for EVERY known trading ticker
3. When whitelisting, websocket connections are opened for EVERY known trading ticker relating symbols on the whitelist

In both cases, we should only open websocket connections for tickers involved in an identified triangular relationship

[bmino]

Pushed my changes to the develop branch but then deleted my API keys and I can't generate new ones... Hopefully tomorrow, the Binance email server will be running again

[dlasher]

This looks better:

Whitelist of 50:

Checking configuration ... Successfully pinged Binance in 188 ms Found 657/920 currently trading tickers Found 50 triangular relationships Opening 82 depth websockets ...

Whitelist of 100:

Checking configuration ... Successfully pinged Binance in 348 ms Found 657/920 currently trading tickers Found 78 triangular relationships Opening 134 depth websockets ...

I'll leave it running passive for a while, see if it's happier.

[dlasher]

Something odd.. no matter what I set the profit margin, or trading costs to, it gives me this at the top: │ BTC-BAND-BNB │ 0.0000% │ 8.00 │ 3.50 │ 8.90 │ 8.90 │ │ BTC-EOS-BNB │ -0.2939% │ 7.30 │ 4.10 │ 8.90 │ 8.90 │

Ignore the second line.. that first one is NOT 0.000% profit, and even if it was, it should change as I adjust the trading fees, right?

(I like seeing AB age, BC age CA age - very cool!)

BAND disappeared after a little while, assuming it was some fleeting weirdness on Binance's part. Ignore this for now, I can't repeat it.. other than the 10 times in a row it did it 20 minutes ago.. .lol

[dlasher]

And starting to see these in the log, new error message ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: ltcbtc@depth@100ms ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: ethbtc@depth@100ms ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: bnbbtc@depth@100ms ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: qtumbtc@depth@100ms ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: bntbtc@depth@100ms ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: linkbtc@depth@100ms ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: trxbtc@depth@100ms ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: nulseth@depth@100ms ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: bandbtc@depth@100ms ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: bchbtc@depth@100ms ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: sxpbtc@depth@100ms ["8/11/2020, 4:40:09 AM"] INFO : Terminating inactive/broken WebSocket: sxpbnb@depth@100ms

They do eventually get followed up with messages like: ["8/11/2020, 4:41:11 AM"] INFO : WebSocket closed: ltcbtc@depth@100ms (1006) ["8/11/2020, 4:41:11 AM"] INFO : WebSocket reconnecting: ltcbtc@depth@100ms... ["8/11/2020, 4:41:11 AM"] INFO : Subscribed to ltcbtc@depth@100ms

So I assume that means network tuning still needed on my side.

[dlasher]

From a CPU standpoint, it's now completely flatlined on one core (didn't do that before) so something is working much harder! 👍

[bmino]

I think that 0% weirdness might happen when a websocket hasn't responded with the first piece of data. I'll check tomorrow

The terminating issue and subsequent reconnecting is caused by our api wrapper and I'm already looking into what causes this

Excellent catch on the excessive socket counts!!

[dlasher]

With no whitelist:

Checking configuration ... Successfully pinged Binance in 152 ms Found 657/920 currently trading tickers Found 161 triangular relationships Opening 288 depth websockets ...

[dlasher]

FYI, even with lots of cores:

["8/10/2020, 10:25:00 PM"] DEBUG: Operating System: Linux ["8/10/2020, 10:25:00 PM"] DEBUG: Cores Speeds: [1531,1464,1445,1457,1447,1455,1427,1437,1432,1444,1830,1989,1418,1461,1441,1413,1417,1455,1420,1462,1439,1460,1423,1515,1460,1492,1427,1439,1420,1481,1437,1454] MHz

On a decent internet connection Testing download speed................................................................................ Download: 114.34 Mbit/s Testing upload speed................................................................................................ Upload: 7.05 Mbit/s

It still can't keep up without whitelists at 100 depth.

==> /usr/src/binance-triangle-arbitrage/logs/binance.log <== ["8/10/2020, 10:27:39 PM"] INFO : WebSocket closed: mcoeth@depth@100ms (1006) ["8/10/2020, 10:27:39 PM"] INFO : WebSocket reconnecting: mcoeth@depth@100ms... ["8/10/2020, 10:27:39 PM"] INFO : Subscribed to mcoeth@depth@100ms ["8/10/2020, 10:27:40 PM"] INFO : WebSocket closed: hivebtc@depth@100ms (1006) ["8/10/2020, 10:27:40 PM"] INFO : WebSocket reconnecting: hivebtc@depth@100ms... ["8/10/2020, 10:27:40 PM"] INFO : Subscribed to hivebtc@depth@100ms ["8/10/2020, 10:27:40 PM"] INFO : WebSocket closed: cmteth@depth@100ms (1006) ["8/10/2020, 10:27:40 PM"] INFO : WebSocket reconnecting: cmteth@depth@100ms... ["8/10/2020, 10:27:40 PM"] INFO : Subscribed to cmteth@depth@100ms ["8/10/2020, 10:27:40 PM"] INFO : WebSocket closed: cosbnb@depth@100ms (1006) ["8/10/2020, 10:27:40 PM"] INFO : WebSocket reconnecting: cosbnb@depth@100ms... ["8/10/2020, 10:27:40 PM"] INFO : Subscribed to cosbnb@depth@100ms ["8/10/2020, 10:27:40 PM"] INFO : WebSocket closed: balbnb@depth@100ms (1006) ["8/10/2020, 10:27:40 PM"] INFO : WebSocket reconnecting: balbnb@depth@100ms... ["8/10/2020, 10:27:40 PM"] INFO : Subscribed to balbnb@depth@100ms ["8/10/2020, 10:27:40 PM"] INFO : WebSocket closed: powreth@depth@100ms (1006) ["8/10/2020, 10:27:40 PM"] INFO : WebSocket reconnecting: powreth@depth@100ms... ["8/10/2020, 10:27:40 PM"] INFO : Subscribed to powreth@depth@100ms ["8/10/2020, 10:27:40 PM"] INFO : WebSocket closed: troybtc@depth@100ms (1006) ["8/10/2020, 10:27:40 PM"] INFO : WebSocket reconnecting: troybtc@depth@100ms... ["8/10/2020, 10:27:40 PM"] INFO : Subscribed to troybtc@depth@100ms

==> /usr/src/binance-triangle-arbitrage/logs/performance.log <== ["8/10/2020, 10:27:40 PM"] WARN : Bid depth (0) too shallow to convert 2.13 MCO to ETH using MCOETH ["8/10/2020, 10:27:41 PM"] WARN : Bid depth (0) too shallow to convert 113 POWR to ETH using POWRETH ["8/10/2020, 10:27:41 PM"] WARN : Ask depth (0) too shallow to convert 0.001 BTC to STORJ using STORJBTC ["8/10/2020, 10:27:42 PM"] WARN : Ask depth (0) too shallow to convert 0.001 BTC to LSK using LSKBTC ["8/10/2020, 10:27:42 PM"] WARN : Bid depth (0) too shallow to convert 819 CMT to ETH using CMTETH ["8/10/2020, 10:27:42 PM"] WARN : Bid depth (0) too shallow to convert 5.93 WAVES to ETH using WAVESETH ["8/10/2020, 10:27:43 PM"] WARN : Ask depth (0) too shallow to convert 0.001 BTC to WAN using WANBTC ["8/10/2020, 10:27:43 PM"] WARN : Ask depth (0) too shallow to convert 0.001 BTC to WAN using WANBTC ["8/10/2020, 10:27:43 PM"] WARN : Ask depth (0) too shallow to convert 0.001 BTC to GNT using GNTBTC ["8/10/2020, 10:27:44 PM"] WARN : Ask depth (0) too shallow to convert 0.001 BTC to DATA using DATABTC ["8/10/2020, 10:27:44 PM"] WARN : Bid depth (0) too shallow to convert 1204 COS to BNB using COSBNB ["8/10/2020, 10:27:44 PM"] WARN : Ask depth (0) too shallow to convert 0.001 BTC to CHZ using CHZBTC ["8/10/2020, 10:27:45 PM"] WARN : Ask depth (0) too shallow to convert 0.001 BTC to TROY using TROYBTC ["8/10/2020, 10:27:45 PM"] WARN : Bid depth (0) too shallow to convert 86 WRX to BNB using WRXBNB ["8/10/2020, 10:27:45 PM"] WARN : Ask depth (0) too shallow to convert 0.001 BTC to HIVE using HIVEBTC ["8/10/2020, 10:27:45 PM"] WARN : Bid depth (96) too shallow to convert 352 MDT to BNB using MDTBNB ["8/10/2020, 10:27:45 PM"] WARN : Ask depth (0) too shallow to convert 0.001 BTC to STMX using STMXBTC ["8/10/2020, 10:27:45 PM"] WARN : Ask depth (0) too shallow to convert 0.001 BTC to STMX using STMXBTC ["8/10/2020, 10:27:45 PM"] WARN : Bid depth (0) too shallow to convert 0.54 BAL to BNB using BALBNB ["8/10/2020, 10:27:45 PM"] WARN : Completed 142/161 (88.2%) calculations in 5602 ms

[bmino]

This terminating error has only been happening recently. It's so odd because I can run the bot for hours typically without issue, and then start up and it happens within minutes.

I believe this related to how ping and pong frames are scheduled for websockets and am investigating

Btw my machine is 4 cores with good wifi so I don't think this is a spec issue

[dlasher]

So with the disclaimer than I'm a rabid consumer of projects, but not a coder myself, I ask the following question:

Would throwing more threaded behavior at this problem help? thread per websocket? thread per 10 sockets?

[bmino]

Hahaha I can respect that!

I don't think more threads will help here, but I haven't tracked down the road cause yet. My suspicion is we need to modify the logic that determines a websockets is "dead"

I've had a death in the family so I won't be back engaged with this until likely next week

[dlasher]

@bmino sorry to hear that.. life > code

we'll pick this up some other time.

[shu-boom]

I am having the same problem. I increased waiting period but it keeps happening. My internet connection is stable, so I don't know if pm2 will be helpful. I would be happy to contribute to this project. I forked it last week and still need some time to read the code and get myself adjusted once I can make it run lol. Is there any quick workaround for this problem? Please let me know thanks

[bmino]

This issue has been address is v5.5.1 Thank you @dlasher for your insight into identifying this!!

We should open a new issue to track the "Terminating inactive/broken" error