"Unwanted data received from uniqueId" after re-subscription

[HMaker]

Hi, I am using a single connection to listen to updates of several markets. When new markets are collected from betfair's Betting API I need to re-subscribe to add that new market to marketIds filter of marketSubscription op. I have the following function that does this re-subscription where self._stream is an instance of BetfairStream

def _resubscribe(self):
    self._stream.subscribe_to_markets(
        market_filter=streaming_market_filter(market_ids=self._market_ids),
        market_data_filter=self._data_filter,
        initial_clk=self._stream.listener.initial_clk,
        clk=self._stream.listener.clk,
        conflate_ms=self._aggregate_ms
    )

So I am sending initial_clk and clk kept by the attached listener, is that right? But after that I start getting WARNING: Unwanted data received from uniqueId: 1, expecting: 2 logs.

[HMaker]

I extended BetfairStream to add

def resubscribe_to_markets(
    self,
    market_filter: dict,
    market_data_filter: dict,
    conflate_ms: int = None,
    heartbeat_ms: int = None,
    segmentation_enabled: bool = True,
) -> int:
    message = {
        "op": "marketSubscription",
        "id": self.listener.stream_unique_id,
        "marketFilter": market_filter,
        "marketDataFilter": market_data_filter,
        "initialClk": self.listener.initial_clk,
        "clk": self.listener.clk,
        "conflateMs": conflate_ms,
        "heartbeatMs": heartbeat_ms,
        "segmentationEnabled": segmentation_enabled,
    }
    self._send(message)
    return self.listener.stream_unique_id

I think betfair expects same id to be sent on resubscription.

[HMaker]

@liampauling that's the right way to resubscribe?

[liampauling]

Streaming is super lightweight so you shouldn't be filtering on marketIds, the only time you should resubscribe is if you need to reconnect after a disconnection. Resubscribing won't work if you are changing the marketFilter, you need to stop and start a new connection (not efficient or recommend)

[HMaker]

By "lighweight" you mean it does not support subscribing to more than 1 market per connection? By lighweight I understand fast and low on resources.

[liampauling]

It can handle thousands of markets, see this quote from the official docs:

Coarse vs Fine Grain Subscriptions It is preferable to use coarse grain subscriptions (subscribe to a super-set) rather than fine grain (specific market ids). If you find yourself frequently changing subscriptions you probably want to find a wider super-set to subscribe to

[HMaker]

I want to subscribe to WIN markets for all races that will start in next X minutes, they don't have a filter that allows me to do it so I can't subscribe to that super-set and need grain subscriptions. I don't want to subscribe to all markets to avoid uneeded overhead.

[liampauling]

The overhead is nothing in comparison to constantly resubscribing which in this case doesn't work and instead needs a new connection / full image each time.

[HMaker]

Betfair's streaming API supports re-subscription as documented by https://support.developer.betfair.com/hc/en-us/articles/360000391532-How-do-you-unsubscribe-from-a-market-using-the-Stream-API- and https://support.developer.betfair.com/hc/en-us/articles/360000391612-Market-Streaming-how-do-I-managed-re-connections- Re-subscription won't work because this library has no support for it?

[HMaker]

Ok I understood it, initalClk and clk only works when same subscription criteria is used, so in a possible reconnection if I change subscription criteria I will lose all updates that happened after disconnection. I am OK with that, I need just last update anyway.

[liampauling]

Yes, resubscription does work but like you have described its a resubscribe to the current market and data filter rather than a change/resubscribe.