Convert External Dataset to HftBacktest Format

[KevinKU22]

Hi @nkaz001 ,

First thank you for creating this insightful repository. I greatly appreciate your work.

I’m currently trying to use my local exchange tick data to backtest the market-making strategy . Could you pls inform me on the on the feasibility of converting using my dataset, and perhaps provide some guidance or tips on how I can convert my sample data (listed below) into the format required by the hftbacktest dataset Thank you

[nkaz001]

First, if you only have BBO data and no market depth data, achieving accurate backtesting results can be challenging, depending on the characteristics of the strategy. If the strategy’s order execution is not heavily dependent on queue position, this may be less of an issue.

The current stable version does not directly support BBO data; you'll need to convert it to L2 market depth data. You can find the basic concept here. Please feel free to reach out if you need any further assistance.

[nkaz001]

For traditional exchanges with regular daily open and close, it might be better to store snapshots directly in the main data file, rather than in a separate initial snapshot file. If the exchange data is complete, processing it completely without relying on natural refresh is better. Please see the CME MBO processing code for guidance.

https://github.com/nkaz001/hftbacktest/blob/master/py-hftbacktest/hftbacktest/data/utils/databento.py https://github.com/nkaz001/hftbacktest/blob/master/examples/Level-3%20Backtesting.ipynb

[KevinKU22]

hi @nkaz001, sorry for the late reply—have been working on to secure a better tick data source. Just to clarify : if using 5-level MBP snapshot data(still no access to MBO data) to generate trades and depth update/delete incremental events using difforderbooksnapshot.py, would you suggest adding a DEPTH_SNAPSHOT_EVENT at the start of the stream (and perhaps DEPTH_CLEAR_EVENT at the end of the steam) ? Sorry if I misunderstood your previous reply. Thanks

[nkaz001]

I don't quite understand what you mean. I think the processed result would look something like the following for a 3-depth market depth snapshot:

bid market at t0 2 at $10 1 at $9 3 at $8 bid market at t1 2 at $9 3 at $8 1 at $7

then, t0, DEPTH_EVENT | BUY_EVENT, $10, 2 t0, DEPTH_EVENT | BUY_EVENT, $9, 1 t0, DEPTH_EVENT | BUY_EVENT, $8, 3 t1, DEPTH_EVENT | BUY_EVENT, $10, 0 t1, DEPTH_EVENT | BUY_EVENT, $9, 2 t1, DEPTH_EVENT | BUY_EVENT, $7, 1

[KevinKU22]

Thanks for the clarification. I think I misunderstood the point 'store snapshots directly in the main data file' which now I think is specific to the MBO dataset case. So in your example we could process the order book depth update(including depth initialization ) directly using DEPTH_EVENT, and then DEPTH_CLEAR_EVENT and DEPTH_SNAPSHOT_EVENT aren't required, correct?

[nkaz001]

Yes, you're correct. As long as the exchange timestamp and the local timestamp are the same for the snapshot event, as shown in the example above with multiple t0 and t1 events, there is no need to use snapshot events (DEPTH_SNAPSHOT_EVENT and DEPTH_CLEAR_EVENT).

[KevinKU22]

Got it, thank you for confirming.