search

nkaz001 / hftbacktest

A high-frequency trading and market-making backtesting tool in Python and Rust, which accounts for limit orders, queue positions, and latencies, utilizing full tick data for trades and order books, with real-world crypto market-making examples for Binance Futures
MIT License
1.78k stars 357 forks source link

I try to use restful api to get depth-1000 data every 8 hours, but failed occasionally #79

Closed LJingnan closed 3 months ago

LJingnan commented 3 months ago

The error is as follow:

WARNING:root:Mismatch on the book. prev_update_id=4630306859680, pu=4630306859680
INFO:root:sending req to https://fapi.binance.com/fapi/v1/depth?symbol=ethusdt&limit=1000&timestamp=1716047999069: "symbol=ethusdt&limit=1000&timestamp=1716047999069"
WARNING:root:Mismatch on the book. prev_update_id=4630306859793, pu=4630306859793
WARNING:root:Mismatch on the book. prev_update_id=4630306862349, pu=4630306862349
INFO:root:sending req to https://fapi.binance.com/fapi/v1/depth?symbol=btcusdc&limit=1000&timestamp=1716047999171: "symbol=btcusdc&limit=1000&timestamp=1716047999171"
INFO:root:sending req to https://fapi.binance.com/fapi/v1/depth?symbol=btcusdt&limit=1000&timestamp=1716047999173: "symbol=btcusdt&limit=1000&timestamp=1716047999173"
WARNING:root:The book is initialized. symbol=ethusdt, prev_update_id=4630307007357
WARNING:root:The book is initialized. symbol=btcusdc, prev_update_id=4630307009982
正在 __keep_alive
WARNING:root:Timed out on request: /v1/depth ("symbol=btcusdt&limit=1000&timestamp=1716047999173"), retrying...
INFO:root:sending req to https://fapi.binance.com/fapi/v1/depth?timestamp=1716048006162: "timestamp=1716048006162"
ERROR:root:Unhandled Error: 400, message='Bad Request', url=URL('https://fapi.binance.com/fapi/v1/depth?timestamp=1716048006162'): Bad Request
ERROR:root:Endpoint was: GET /v1/depth: "timestamp=1716048006162"

It seems that it is a internet error, but It happens occasionally and unpredictably, which cause my program failed and stop. Do you have any idea to solve this problem. Thank you.

nkaz001 commented 3 months ago

Are you referring to the Python version of collect-binancefutures? If so, please use the collect-binancefutures issue tracker.

Sometimes update id gaps occur repeatedly in the short term, so it may be better to maintain the order book based solely on the diff. depth stream. Retrying to recover by fetching a snapshot delays order book reconstruction and is also limited by API rate limits. The Rust version of collect-binancefutures removes that part and will be provided as an option.

LJingnan commented 3 months ago

Yes I am using the python api. Sorry I should put this in collect-binancefutures issue tracker. "The Rust version of collect-binancefutures removes that part", do you mean that rust version would not try to recover by fetching a snapshot and just stop when there is a Internet error causing data missing.

nkaz001 commented 3 months ago

Precisely, when mismatch of update id is detected, the Python version tries to recover by matching update ids from the fetched snapshot. Until recovery is complete, the depth is not reconstructed(not stored). In contrast, the Rust version fetches the snapshot but does not wait for matching update ids. You can choose how to deal with missing update ids when processing stored raw stream data, but recovery can fail. relying on natural refresh is recommended.

LJingnan commented 3 months ago

Precisely, when mismatch of update id is detected, the Python version tries to recover by matching update ids from the fetched snapshot. Until recovery is complete, the depth is not reconstructed(not stored). In contrast, the Rust version fetches the snapshot but does not wait for matching update ids. You can choose how to deal with missing update ids when processing stored raw stream data, but recovery can fail. relying on natural refresh is recommended.

I see, thank you so much.