betcode-org / betfair

betfairlightweight - Betfair API-NG python wrapper (with streaming)
MIT License
428 stars 147 forks source link

Log warning on unexpected conflation in stream #582

Open agberk opened 3 weeks ago

agberk commented 3 weeks ago

From https://betfair-developer-docs.atlassian.net/wiki/spaces/1smk3cen4v3lu3yomq5qye0ni/pages/2687396/Exchange+Stream+API:

Conflation

Conflation set to true (con=true) in the stream message means that multiple stream updates have been pushed in the same cycle.

This can happen due to the following reasons:

The client socket buffer for the connection needs to be read (cleared) by the client in order for the Stream API to push the next cycle, if not the current update is skipped, and pushed with the next cycle resulting in "con = true".

The client subscription message has the "conflateMs" parameter set to a value greater than '0'.

The Stream API has a slow publishing cycle resulting in multiple updates being pushed in the same message.

I'd like to be able to identify the cases where either the client isn't reading faster enough or the Stream API has a slow publishing cycle.

It's been a while since I've looked at the code but it seemed like the most sensible approach was to modify the BaseStream.on_update method, check whether con has been set to True and log a warning if the conflateMs we asked for is not >0, similar to how the max latency check and warning log statement is being done.

agberk commented 3 weeks ago

PR: https://github.com/betcode-org/betfair/pull/583