Binance futures now has stream limit 200

[OnlyC]

Describe the bug Binance futures now has 200 stream limit (2020-10-27) https://binance-docs.github.io/apidocs/futures/en/#change-log

To Reproduce Use pairs=binance_futures_pairs() the app will freeze without getting any update. I have to make an array of SYMBOLS and remove some from the full list. Full list has 216 symbols.

[GoldenNaim]

Hello,

I have just trying to reproduce the bug, it works fine for me, binance_futures_pairs() return all pairs with this simple code :

from decimal import Decimal
from cryptofeed.exchanges import BinanceFutures
from cryptofeed.pairs import binance_futures_pairs
def main():
    pairs=binance_futures_pairs()
    print(pairs)

if __name__ == '__main__':
    main()

[OnlyC]

I mean it will freeze when you use in like: add_feed(pairs=binance_futures_pairs()) & feed.run().

[GoldenNaim]

Okay, I will try to see what happened my crypto-friend

[GoldenNaim]

It works fine for me with this code :

from decimal import Decimal
from cryptofeed import FeedHandler
from cryptofeed.callback import LiquidationCallback, OpenInterestCallback, TradeCallback 
from cryptofeed.defines import BID, ASK, LIQUIDATIONS, OPEN_INTEREST, TRADES
from cryptofeed.exchanges import BinanceFutures
from cryptofeed.pairs import binance_futures_pairs

async def oi(feed, pair, open_interest, timestamp, receipt_timestamp):
    print(f'Timestamp: {timestamp} Feed: {feed} Pair: {pair} open interest: {open_interest}')

def main():
    f = FeedHandler()
    f.add_feed(BinanceFutures(pairs=binance_futures_pairs(), channels=[OPEN_INTEREST], callbacks={OPEN_INTEREST: OpenInterestCallback(oi)}))
    f.run()

if __name__ == '__main__':
    main()

[OnlyC]

Please try with 4 channels: TRADES, L2_BOOK, OPEN_INTEREST, LIQUIDATIONS

[GoldenNaim]

Okay, it seems the bug it's only with the LiquidationCallback

[OnlyC]

try with 3 channels also freeze [TRADES, L2_BOOK, TICKER] 74 pairs * 3 = 222 streams > 200 streams limit.

[GoldenNaim]

But works fine with : channels=[OPEN_INTEREST, TRADES, L2_BOOK] .

I think the problem is not here

[OnlyC]

The OPEN_INTEREST is not a stream, it's a restAPI call.

[GoldenNaim]

You have right, I will try something to see if it works, 5min.

[GoldenNaim]

Okay so, the Binance documentation say : The maximum stream number that a single connection can listen to changes as 200.

The solution is : create multiple add_feed to not exceed 200 by single connection.

So this solution work fine for me ( tested ) :

def main():
    f = FeedHandler()
    f.add_feed(BinanceFutures(pairs=binance_futures_pairs(), channels=[OPEN_INTEREST, TRADES, L2_BOOK], callbacks={OPEN_INTEREST: OpenInterestCallback(oi), TRADES: TradeCallback(trade), L2_BOOK: BookCallback(book)}))
    f.add_feed(BinanceFutures(pairs=binance_futures_pairs(), channels=[LIQUIDATIONS], callbacks={LIQUIDATIONS: LiquidationCallback(liquidations)}))
    f.add_feed(BinanceFutures(pairs=binance_futures_pairs(), channels=[TICKER], callbacks={TICKER: TickerCallback(ticker)}))
    f.run()

Full code for test :

from decimal import Decimal
from cryptofeed import FeedHandler
from cryptofeed.callback import LiquidationCallback, OpenInterestCallback, TradeCallback, BookCallback, TickerCallback
from cryptofeed.defines import BID, ASK, LIQUIDATIONS, OPEN_INTEREST, TRADES, L2_BOOK, TICKER
from cryptofeed.exchanges import BinanceFutures
from cryptofeed.pairs import binance_futures_pairs

async def oi(feed, pair, open_interest, timestamp, receipt_timestamp):
    print(f'FEED OPEN_INTEREST')

async def ticker(feed, pair, bid, ask, timestamp, receipt_timestamp):
    print(f'FEED TICKER ')

async def liquidations(feed, pair, side, leaves_qty, price, order_id, timestamp, receipt_timestamp):
    print(f"FEED LIQUIDATIONS")

async def trade(feed, pair, order_id, timestamp, side, amount, price, receipt_timestamp):
    assert isinstance(timestamp, float)
    assert isinstance(side, str)
    assert isinstance(amount, Decimal)
    assert isinstance(price, Decimal)
    print(f"FEED TRADE")

async def book(feed, pair, book, timestamp, receipt_timestamp):
    print(f'FEED L2_BOOK')

def main():
    f = FeedHandler()
    f.add_feed(BinanceFutures(pairs=binance_futures_pairs(), channels=[OPEN_INTEREST, TRADES, L2_BOOK], callbacks={OPEN_INTEREST: OpenInterestCallback(oi), TRADES: TradeCallback(trade), L2_BOOK: BookCallback(book)}))
    f.add_feed(BinanceFutures(pairs=binance_futures_pairs(), channels=[LIQUIDATIONS], callbacks={LIQUIDATIONS: LiquidationCallback(liquidations)}))
    f.add_feed(BinanceFutures(pairs=binance_futures_pairs(), channels=[TICKER], callbacks={TICKER: TickerCallback(ticker)}))
    f.run()

if __name__ == '__main__':
    main()

[OnlyC]

Ahh, nice. Thank you!

[OnlyC]

Then this issue shouldn't tag as a bug, I can't change it.

[bmoscon]

so I'm confused - is there an issue still or not?

[OnlyC]

Yes, there is an issue with Binance that we can't use 1 feed connection to stream all the pairs but we can easily solve it with more feeds. I think a notice in the docs will do the job.

[bmoscon]

let me see if its possible to get something working in the code that solves this.

[davidtwomey]

@bmoscon note this also affects Binance spot (BINANCE), as well as BINANCE_FUTURES

BINANCE (Spot) docs under websocket limits:

"A single connection can listen to a maximum of 1024 streams"

BINANCE (Futures) docs

"The maximum stream number that a single connection can listen to changes as 200." (also shows up in change log on the same page for 2020-Oct-27)

Note: this only appears in the tether USDT margined futures but not the COIN margin futures...but suspect the limit probably exists on both.

If i can find some time i will test empircally, but just from a quick test Binance SPOT TRADES channel seems to hit a limit when trying to subscribe to 965 symbols. However, the Binance SPOT TICKER channel does not...:sigh

[bmoscon]

I have a fix for this, but it will require waiting until I've rolled out my update to the connection architecture to allow exchanges to connect on more than one websocket. Probably another week or two

[bmoscon]

this has been fixed!