search

abbass2 / pyqstrat

A fast, extensible, transparent python library for backtesting quantitative strategies.
BSD 3-Clause "New" or "Revised" License
353 stars 57 forks source link

too much data for in memory #28

Closed Macfly closed 9 months ago

Macfly commented 1 year ago

what is the best approach when data is too big to fit in memory? like when running an option strat on 1000 stocks for multiple years.

abbass2 commented 9 months ago

Each indicator needs a column in memory. But the indicator can be an array of booleans which should not take much less memory than a full dataframe with many columns. Each indicator is the length of the timestamps you use for evaluation, so if you have minute bars for 10 years, that should be an array with 5 million entries. The signals and rules can load data on demand from, for example, a hdf5 file. If you want to give a more detailed example of the problem you are solving, I could give you more ideas.