Reduced conversion time by loading input file into memory once for

FX31337 / FX-BT-Scripts

:page_facing_up: Useful scripts for backtesting.

MIT License

34 stars 39 forks source link

Reduced conversion time by loading input file into memory once for #51

Closed Kostafun closed 8 years ago

Kostafun commented 8 years ago

all timeframes. Lots of CPU cycles were wasted on converting time field in every tick in input file, separately for each timeframe.

kenorb commented 8 years ago

I'm sorry, but these commits from PR #47 started before ~13h ago, so I had to merge. Could you rebase your code and append relevant parts again?

Kostafun commented 8 years ago

Sure :) My current fix is very fast (6-7 times faster than base in my tests), but could crash on machines with not enough RAM or on a very large datasets. I'll commit updated fix for this issue (with numpy.memmap temporary file holding input data) - it should be as fast when there are enough RAM, and with temp file in case of low RAM. I'll finish and commit it tomorrow.

kenorb commented 8 years ago

I had to reset the master to the previous state and moved the other changes into performance branch, as it wasn't ready yet. I'll test that out and see if I can merged it first.

kenorb commented 8 years ago

CI test results based on the 1 month data conversion: Before: Elapsed time 24 min 44 sec After: Elapsed time 12 min 56 sec