search

nkaz001 / hftbacktest

A high-frequency trading and market-making backtesting tool in Python and Rust, which accounts for limit orders, queue positions, and latencies, utilizing full tick data for trades and order books, with real-world crypto market-making examples for Binance Futures
MIT License
1.78k stars 357 forks source link

Avoid incomplete npz file when force end histfile conversion #108

Closed WannabeBotter closed 1 month ago

WannabeBotter commented 1 month ago

After my Bybit histdata utility is merged, I realized when I force stop the convert process, incomplete npz file may remain. So I changed the utility so that it won't leave incomplete npz file.

nkaz001 commented 1 month ago

Does that happen only if you force stop the process or an error occurs? If so, I don't think we need to handle that situation.

WannabeBotter commented 1 month ago

You are right. The issue occurs only when the user forcefully stops the conversion process. I found the issue when I convert many files into npz format from a script.

In such situation, there is a workaround to deal with the issue by removing the generated npz files with the latest timestamps.

nkaz001 commented 1 month ago

A user needs to write at least a short Python script to convert the data using convert method. If a user needs to specifically handle such a situation, they should retrieve the data without output_filename argument and then process it as you described. In my opinion, the library function does not need to handle such situations internally.

WannabeBotter commented 1 month ago

Understood. Closing this pull request.