RealJohnGalt
commented
9 years ago Hi Alan, Thanks for looking at the software. Some of this is due to parts of the software not being upgraded with the rest. Early on when there were fewer indicators and more simple TradeIndicator handling, much of the intent was to have all indicators run constantly for custom strategies to be developed in strategies.py. Some indicators also depend on each other for simpler calculations (e.g. DEMA on EMA, *StochRSI on RSI). I'm currently working on a set of changes which imperfectly solve this issue. Here are the planned/developed changes:
- Add new objects to each indicator class in indicators.py:
- IndicatorDepends: list of strings of indicators which are depended on (if applicable).
- CandleDepends: number of candles which are needed to get results with current configuration.
- Add a new argument to storage.writelist for how many objects should be held in the each list.
- Only use TradeIndicators and IndicatorDepends for running indicators.
- Support printing number of required candles before getting enough info/trading (uses CandleDepends).
- Using per-indicator shelve files with all genconfig indicator class objects included in the name. This fixes a bug if the software is resumed with changed configuration. This also avoids the need for a queue while again using concurrent futures for running indicators.
- Only store the max CandleDepends number of candles in sqlite. While generally only the last candle is needed once the indicator is producing output, this is a far simpler approach and still keeps size down tremendously.
I've noticed that there is no limit to the amount of past data kept in both the SQLite and shelve databases. Eventually an error about page size, and other shelve read/write errors are thrown - at this point I saw that the *.shelve.db file was over 2 GB. This was an extreme case with short candles so it grew pretty fast, but it's an issue for long runs regardless.
As a quick fix you could splice off the end of the list in the
writelistfunction, i.e.temp = temp[-storageMax:]. It would be nice if it was possible to set thestorageMaxbased on what's needed by the config.I don't have a one-liner fix for the SQLite table, and haven't encountered problems yet, but it is important to look at since the entire market history is being loaded every interval.
As a side note, it seems every indicator is being calculated and stored, even if I'm just using one.
I hope to find time to contribute to the code in the near future, finding my way around it now.
-alanfuji