search

FX31337 / FX-BT-Scripts

:page_facing_up: Useful scripts for backtesting.
MIT License
34 stars 39 forks source link

convert_csv_to_mt.py: Further performance improvements for conversion. [$40 awarded] #59

Closed kenorb closed 8 years ago

kenorb commented 8 years ago

Follow-up to #37

Currently the script uses mmap to read the file in memory to leave some part of the work to the OS, but that's just a marginal speedup, the real one is in the optimized timestamp parser and the more compact pack. It might be possible to shave some more time by shuffling some code around though.

--- The **[$40 bounty](https://www.bountysource.com/issues/32438471-convert_csv_to_mt-py-further-performance-improvements-for-conversion?utm_campaign=plugin&utm_content=tracker%2F20487492&utm_medium=issues&utm_source=github)** on this issue has been claimed at [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F20487492&utm_medium=issues&utm_source=github).
kenorb commented 8 years ago

Seems to work.

Old build:

HST: 27 min FXT: 40 min Total time: 1 hr 8 min

New build:

HST: 10 min FXT: 21 min Total time: 31 min

Since I've implemented a split per format, I'll leave it like that for now, it makes more sense.