search

Ameobea / algobot3

Third iteration of the algorithmic trading bot I've been working on.
5 stars 0 forks source link

Severe issues with mongodb performance #2

Closed Ameobea closed 8 years ago

Ameobea commented 8 years ago

As backtests run for more than a few minutes, mongodb becomes incredibly slow and symptoms of overload become apparent such as system load reaching upwards of 200.

I believe this is due to poor indexing in the database as the database size increases, but I'm not 100% sure. If that is the case, it will absolutely be necessary to incorporate some kind of local cache for very highly regularly accessed data, namely ticks. That will probably be a good idea either way, as I predict the majority of calculations being made by the bot to be on rather short timeframes.

dalexj commented 8 years ago

just use postgres™

Ameobea commented 8 years ago

A cache for ticks/averages/momentums should be maintained within the application in order to reduce the amount of queries made to the database.

Due to recently added tick -> period data conversions for storage, during normal bot operation it is unlikely/impossible that the database will become overloaded. However, for backtesting strategies as fast as possible it is necessary to push hardware to the limit, avoiding as many bottlenecks as possible.

Perhaps create an advanced backtesting mode that pre-generates data such as smas and momentums to be used by the strategy.