search

mementum / bta-lib

Technical Analysis library in pandas for backtesting algotrading and quantitative analysis
MIT License
454 stars 106 forks source link

Memory Usage when calling e.g. btalib.rsi(df) in a loop #11

Open moritzgun opened 3 years ago

moritzgun commented 3 years ago

I am running a MultiplexSocket on Binance API to get 5 minute klines of around 200 symbols. On every new completed candle I call a "run_technical_analysis" function. Inside that function I calculate indicators for each symbol like this:

period = 14 value = btalib.ema(candlesticks_df.tail(period + 1), period=period) ## Version A

value = candlesticks_df.Close.tail(14).ewm(span=14).mean() ## Version B

After that I append value.df.tail(1) to a list. On Version A my RAM goes up almost 10 MB every 5 minutes. It seems like the allocated memory for 'value' is not beeing freed. When I use Version B instead there is no such issue.

What can I do to reduce that memory usage?

reedajohns commented 3 years ago

Same problem on my end!

drhighliner commented 3 years ago

Same problem on my end as well. Does anybody have a fix for this? Maybe we can create a fork and publish it on pypi ... seems like the developer is not updating this library anymore. :(

PapoKarlo commented 3 years ago

with btalib.stochastic(df) same problem, memory leak very fast.

sam1946 commented 3 years ago

https://stackoverflow.com/questions/13784192/creating-an-empty-pandas-dataframe-then-filling-it

This may solve the issue? Never append a row to a dataframe

Davesmacer commented 2 years ago

Every indicator with btalib inside a loop is filling memory, even if we use del to free variables returned. This is a big issue.