search

nkaz001 / hftbacktest

A high-frequency trading and market-making backtesting tool in Python and Rust, which accounts for limit orders, queue positions, and latencies, utilizing full tick data for trades and order books, with real-world crypto market-making examples for Binance Futures
MIT License
1.78k stars 357 forks source link

Bybit histdata #99

Closed WannabeBotter closed 1 month ago

WannabeBotter commented 1 month ago

Added a utility function to convert ByBit public historical data files (zip for orderbook and csv.gz for trades) to hftbacktest format.

The supported public historical data can be downloaded from https://www.bybit.com/derivatives/en/history-data

nkaz001 commented 1 month ago

Thanks for your contribution!

Could you add a DEPTH_CLEAR_EVENT before the snapshot event, specifying the price up to which the market depth needs to be cleared? It's not strictly necessary but helpful for maintaining more accurate market depth information.

By the way, do you know which topic is provided for order book data regarding the depth level? Bybit provides different update frequencies according to the depth level.

WannabeBotter commented 1 month ago

Thank you for your quick review and feedback. I added DEPTH_CLEAR_EVENT. Please take a look.

The topic in ByBit public orderbook historical data is ob500. (L2 orderbook) The update frequency is every 100ms as described in the ByBit API V5 document.

Each file represents a day in UTC. Each file includes at least two snapshots. One is a snapshot at the beginning of the day and another is a snapshot at the beginning of the next day.

nkaz001 commented 1 month ago

Shouldn't it be size instead of homeNotional at https://github.com/nkaz001/hftbacktest/pull/99/files#diff-a58bc89be6cc8787d661c0f92150fd76008d8b661fbbb341ebd591dacc2577fbR136

WannabeBotter commented 1 month ago

Thank you for your review. I replaced homeNotional with size and reflected to my branch.

In Bybit historical trades CSV, size and homeNotional are always the same value. So this change doesn't have any practical difference. But I agree that size is easier to understand than homeNotional for the future readers.

size : the trade amount represented by the base currency homeNotional : the same as size foreignNotional: the trade amount represented by the quote currency

Here is an example of Bybit trades historical data CSV.

timestamp,symbol,side,size,price,tickDirection,trdMatchID,grossValue,homeNotional,foreignNotional 1703980800.2195,ETHUSDT,Sell,0.42,2293.30,MinusTick,b09d8a8b-9b56-5ead-8ee3-71100edb823f,9.63186e+10,0.42,963.186 1703980800.2195,ETHUSDT,Sell,1,2293.30,ZeroMinusTick,28c8037b-7199-520d-9e4a-8c058a15b574,2.2933000000000003e+11,1,2293.3

nkaz001 commented 1 month ago

Thank you for your contribution!