Error with df.ta.strategy("All")

[Fredrik81]

Which version are you running? The lastest version is on Github. Pip is for major releases.

Python: 3.8.6
pandas-ta: 0.2.23b0

Upgrade.

done

Describe the bug A clear and concise description of what the bug is.

These examples fails on the latest version:

The Default Strategy is the ta.AllStrategy. The following are equivalent:

df.ta.strategy() df.ta.strategy("All") df.ta.strategy(ta.AllStrategy)

Specifying specific categories work (['candles', 'momentum', 'overlap', 'performance', 'statistics', 'trend', 'volatility', 'volume'])

These have no effect on the result: dataframe.ta.mp = True dataframe.ta.mp = False

Error:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/opt/tradebot/freqtrade/.env/lib/python3.8/site-packages/pandas_ta/core.py", line 432, in _mp_worker
    return getattr(self, method)(*args, **kwargs)
TypeError: crossed() got an unexpected keyword argument 'append'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./.env/lib/python3.8/site-packages/pandas_ta/core.py", line 682, in strategy
    [self._post_process(r, **kwargs) for r in results]
  File "./.env/lib/python3.8/site-packages/pandas_ta/core.py", line 682, in <listcomp>
    [self._post_process(r, **kwargs) for r in results]
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 448, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
TypeError: crossed() got an unexpected keyword argument 'append'

To Reproduce Download sample data and try the above examples.

Expected behavior All indicators added to dataframe.

Thanks for using Pandas TA! Thanks for a great TA library!

[twopirllc]

Hello @Fredrik81,

The Latest Version is the Github version 0.2.42b0. You have the last stable version 0.2.23b0 from pypi.

Please upgrade and give it another shot. If it still does not work, we can go from there.

$ pip install -U git+https://github.com/twopirllc/pandas-ta

Kind Regards, KJ

[Fredrik81]

Same issue :-(

pip install -U git+https://github.com/twopirllc/pandas-ta
Collecting git+https://github.com/twopirllc/pandas-ta
  Cloning https://github.com/twopirllc/pandas-ta to /tmp/pip-req-build-72f7uy_2
  Running command git clone -q https://github.com/twopirllc/pandas-ta /tmp/pip-req-build-72f7uy_2
Requirement already satisfied: pandas in ./.env/lib/python3.8/site-packages (from pandas-ta==0.2.42b0) (1.2.2)
Requirement already satisfied: pytz>=2017.3 in ./.env/lib/python3.8/site-packages (from pandas->pandas-ta==0.2.42b0) (2020.1)
Requirement already satisfied: numpy>=1.16.5 in ./.env/lib/python3.8/site-packages (from pandas->pandas-ta==0.2.42b0) (1.20.1)
Requirement already satisfied: python-dateutil>=2.7.3 in ./.env/lib/python3.8/site-packages (from pandas->pandas-ta==0.2.42b0) (2.8.1)
Requirement already satisfied: six>=1.5 in ./.env/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas->pandas-ta==0.2.42b0) (1.15.0)

$ python
Python 3.8.6 (default, Sep 25 2020, 09:36:53)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas_ta as ta
>>> ta.version
'0.2.42b0'

[twopirllc]

@Fredrik81,

Hmmm... can you provide more reproducible code with sample data?

Thanks, KJ

[Fredrik81]

@twopirllc Hmm i did a smaller manual script and it worked:

import ccxt
import pandas_ta as ta
import pandas as pd

exchange = ccxt.binance()

candles = exchange.fetch_ohlcv('BTC/USDT', timeframe='5m')

df = pd.DataFrame(candles, columns=['date', 'open', 'high', 'low', 'close', 'volume'])
df['date'] = pd.to_datetime(df['date'], unit='ms')
df = df.set_index('date')

df.ta.strategy()
print(df.tail(5))

So i could use some help at least for troubleshooting :-) thinking the secret is in that one of the indicators might not be in the categories? ['candles', 'momentum', 'overlap', 'performance', 'statistics', 'trend', 'volatility', 'volume'] As all of these are working if i add them one by one.. what indicator am i missing?

[twopirllc]

@Fredrik81,

Thanks for giving me something to work with. It is weird because you are getting a TypeError: crossed() got an unexpected keyword argument 'append'. There is no crossed() in core.py that can be called. Anyhow, I will take a look at the code.

Thanks, KJ

[twopirllc]

@Fredrik81,

I had no problem with that script nor using df.ta.strategy("all"). I need some code that breaks.

Thanks, KJ

[Fredrik81]

I use it in combination to freqtrade platform.

        print(type(dataframe))
        print('-------------------')
        print(dataframe.dtypes)
        print('-------------------')
        print(dataframe.tail(1))
        print('-------------------')
        dataframe.ta.strategy()

Return:

<class 'pandas.core.frame.DataFrame'>
-------------------
date      datetime64[ns, UTC]
open                  float64
high                  float64
low                   float64
close                 float64
volume                float64
dtype: object
-------------------
                           date     open     high      low    close    volume
12116 2021-02-12 00:00:00+00:00  27.8645  27.9951  27.7551  27.7933  51591.83
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "./freqtrade/.env/lib/python3.8/site-packages/pandas_ta/core.py", line 432, in _mp_worker
    return getattr(self, method)(*args, **kwargs)
TypeError: crossed() got an unexpected keyword argument 'append'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./freqtrade/freqtrade/strategy/interface.py", line 668, in ohlcvdata_to_dataframe
    return {pair: self.advise_indicators(pair_data.copy(), {'pair': pair})
  File "./freqtrade/freqtrade/strategy/interface.py", line 668, in <dictcomp>
    return {pair: self.advise_indicators(pair_data.copy(), {'pair': pair})
  File "./freqtrade/freqtrade/strategy/interface.py", line 685, in advise_indicators
    return self.populate_indicators(dataframe, metadata)
  File "./freqtrade/user_data/strategies/Indicator.py", line 365, in populate_indicators
    dataframe.ta.strategy()
  File "./freqtrade/.env/lib/python3.8/site-packages/pandas_ta/core.py", line 682, in strategy
    [self._post_process(r, **kwargs) for r in results]
  File "./freqtrade/.env/lib/python3.8/site-packages/pandas_ta/core.py", line 682, in <listcomp>
    [self._post_process(r, **kwargs) for r in results]
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 448, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
TypeError: crossed() got an unexpected keyword argument 'append'

[twopirllc]

@Fredrik81,

That is weird. There is no crossed() in Pandas TA but somehow FreqTrade might be injecting into the dataframe before dataframe.ta.strategy() is called. 🤷🏼‍♂️

Could you add the following print statement to your local Pandas TA ./freqtrade/.env/lib/python3.8/site-packages/pandas_ta/core.py. I want to see if crossed() shows up in the multiprocessing pool somehow like the error suggests. If it does, please give me the full printed list.

    def _mp_worker(self, arguments: tuple):
        """Multiprocessing Worker to handle different Methods."""
        method, args, kwargs = arguments

        print(f"{method}({args}, {kwargs})") # Add this line
        if method != "ichimoku":
            return getattr(self, method)(*args, **kwargs)
        else:
            return getattr(self, method)(*args, **kwargs)[0]

Also can you share more of the file: _"./freqtrade/userdata/strategies/Indicator.py" please? Perhaps something else is occurring that we are not seeing. Are you also importing qtpylib or talib in Indicator.py?

Thanks, KJ

[Fredrik81]

@twopirllc Here are the output from the print:

-------------------
aberration((), {'append': True})
apo((), {'append': True})
cci((), {'append': True})
cmf((), {'append': True})
dema((), {'append': True})
eri((), {'append': True})
aroon((), {'append': True})
hma((), {'append': True})
donchian((), {'append': True})
kc((), {'append': True})
cmo((), {'append': True})
coppock((), {'append': True})
dpo((), {'append': True})
fisher((), {'append': True})
efi((), {'append': True})
ema((), {'append': True})
entropy((), {'append': True})
eom((), {'append': True})
er((), {'append': True})
atr((), {'append': True})
macd((), {'append': True})
mad((), {'append': True})
fwma((), {'append': True})
ha((), {'append': True})
cdl_doji((), {'append': True})
cdl_inside((), {'append': True})
crossed((), {'append': True})
midprice((), {'append': True})
mom((), {'append': True})
accbands((), {'append': True})
natr((), {'append': True})
ad((), {'append': True})
adosc((), {'append': True})
massi((), {'append': True})
adx((), {'append': True})
mcgd((), {'append': True})
kdj((), {'append': True})
keltner_channel((), {'append': True})
pgo((), {'append': True})
bbands((), {'append': True})
bias((), {'append': True})
bollinger_bands((), {'append': True})
pvt((), {'append': True})
pwma((), {'append': True})
qqe((), {'append': True})
cfo((), {'append': True})
nvi((), {'append': True})
amat((), {'append': True})
ppo((), {'append': True})
psar((), {'append': True})
ao((), {'append': True})
aobv((), {'append': True})
obv((), {'append': True})
hull_moving_average((), {'append': True})
rolling_max((), {'append': True})
rvgi((), {'append': True})
squeeze((), {'append': True})
rvi((), {'append': True})
ohlc4((), {'append': True})
pdist((), {'append': True})
percent_return((), {'append': True})
tdi((), {'append': True})
typical_price((), {'append': True})
wcp((), {'append': True})
weighted_bollinger_bands((), {'append': True})
zlwma((), {'append': True})
heikinashi((), {'append': True})
psl((), {'append': True})
ssf((), {'append': True})
pvi((), {'append': True})
pvo((), {'append': True})
pvol((), {'append': True})
pvr((), {'append': True})
session((), {'append': True})
median((), {'append': True})
mfi((), {'append': True})
mid_price((), {'append': True})
stdev((), {'append': True})
stoch((), {'append': True})
stochrsi((), {'append': True})
supertrend((), {'append': True})
swma((), {'append': True})
t3((), {'append': True})
cg((), {'append': True})
chop((), {'append': True})
chopiness((), {'append': True})
qstick((), {'append': True})
quantile((), {'append': True})
returns((), {'append': True})
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "./freqtrade/.env/lib/python3.8/site-packages/pandas_ta/core.py", line 432, in _mp_worker
    return getattr(self, method)(*args, **kwargs)
TypeError: crossed() got an unexpected keyword argument 'append'
"""

Here is the function after i edit.

    def _indicators_by_category(self, name: str) -> list:
        """Returns indicators by Categorical name."""
        return Category[name] if name in self.categories else None

    def _mp_worker(self, arguments: tuple):
        """Multiprocessing Worker to handle different Methods."""
        method, args, kwargs = arguments
        print(f"{method}({args}, {kwargs})") # Add this line
        if method != "ichimoku":
            return getattr(self, method)(*args, **kwargs)
        else:
            return getattr(self, method)(*args, **kwargs)[0]

    def _post_process(self, result, **kwargs) -> Tuple[pd.Series, pd.DataFrame]:

Yes i'm importing both qtpylib and talib but did a quick test to remove them and it did not change anything.

Here are the complete imports

# --- Do not remove these libs ---
import numpy as np
from numpy.lib.function_base import append  # noqa <- removed this as it was not needed and it made no difference
import pandas as pd  # noqa
from pandas import DataFrame

from freqtrade.strategy.interface import IStrategy

# --------------------------------
# Add your lib to import here
#import talib.abstract as ta
import pandas_ta as pta
#import freqtrade.vendor.qtpylib.indicators as qtpylib

[twopirllc]

@Fredrik81,

Yes i'm importing both qtpylib and talib but did a quick test to remove them and it did not change anything.

That's odd. Well somehow qtpylib, though not imported, is appending it's indicators to the DataFrame.

Qtpylib Indicators

In your Indicator.py code, could you add:

    print(type(dataframe))
    print('-------------------')
    print(dataframe.dtypes)
    print('-------------------')
    print(dataframe.tail(1))
    print('-------------------')
    ta_indicators = list((x for x in dir(dataframe.ta) if not x.startswith("_") and not x.endswith("_")))
    print(", ".join(ta_indicators))
    # dataframe.ta.strategy()

When you get a chance, could you please share the output for both cases:

1. importing both qtpylib and pandas_ta
2. only importing pandas_ta.

Thanks, KJ

[Fredrik81]

@twopirllc Sure :-) Here is the code:

        dataframe.ta.mp = True
        #print(dataframe.columns)
        #print(dataframe.ta.adx().tail(2))
        print(type(dataframe))
        print('-------------------')
        print(dataframe.dtypes)
        print('-------------------')
        print(dataframe.tail(1))
        print('-------------------')
        ta_indicators = list((x for x in dir(dataframe.ta) if not x.startswith("_") and not x.endswith("_")))
        print(", ".join(ta_indicators))

Without qtpylib:

# --- Do not remove these libs ---
import numpy as np
#from numpy.lib.function_base import append  # noqa
import pandas as pd  # noqa
from pandas import DataFrame

from freqtrade.strategy.interface import IStrategy

# --------------------------------
# Add your lib to import here
#import talib.abstract as ta
import pandas_ta as pta
#import freqtrade.vendor.qtpylib.indicators as qtpylib

Output:

<class 'pandas.core.frame.DataFrame'>
-------------------
date      datetime64[ns, UTC]
open                  float64
high                  float64
low                   float64
close                 float64
volume                float64
dtype: object
-------------------
                           date     open     high      low    close    volume
12116 2021-02-12 00:00:00+00:00  27.8645  27.9951  27.7551  27.7933  51591.83
-------------------
aberration, above, above_value, accbands, ad, adjusted, adosc, adx, amat, ao, aobv, apo, aroon, atr, bbands, below, below_value, bias, bollinger_bands, bop, brar, categories, cci, cdl_doji, cdl_inside, cfo, cg, chop, chopiness, cksp, cmf, cmo, constants, coppock, cores, cross, cross_value, crossed, crossed_above, crossed_below, datetime_ordered, decay, decreasing, dema, donchian, dpo, efi, ema, entropy, eom, er, eri, fisher, fwma, ha, heikinashi, hilo, hl2, hlc3, hma, hull_moving_average, ibs, ichimoku, implied_volatility, increasing, indicators, inertia, kama, kc, kdj, keltner_channel, kst, kurtosis, linreg, log_return, log_returns, long_run, macd, mad, massi, mcgd, median, mfi, mid_price, midpoint, midprice, mom, mp, natr, nvi, obv, ohlc4, pdist, percent_return, pgo, ppo, psar, psl, pvi, pvo, pvol, pvr, pvt, pwma, qqe, qstick, quantile, returns, reverse, rma, roc, rolling_max, rolling_mean, rolling_min, rolling_std, rolling_vwap, rolling_weighted_mean, rsi, rsx, rvgi, rvi, session, short_run, sinwma, skew, slope, sma, smi, squeeze, ssf, stdev, stoch, stochrsi, strategy, supertrend, swma, t3, tdi, tema, thermo, trend_return, trima, trix, true_range, tsi, ttm_trend, typical_price, ui, uo, variance, version, vidya, vortex, vp, vwap, vwma, wcp, weighted_bollinger_bands, willr, wma, zlema, zlhma, zlma, zlsma, zlwma, zscore

With qtpylib:

# --- Do not remove these libs ---
import numpy as np
#from numpy.lib.function_base import append  # noqa
import pandas as pd
from pandas import DataFrame

from freqtrade.strategy.interface import IStrategy

# --------------------------------
# Add your lib to import here
#import talib.abstract as ta
import pandas_ta as pta
import freqtrade.vendor.qtpylib.indicators as qtpylib

<class 'pandas.core.frame.DataFrame'>
-------------------
date      datetime64[ns, UTC]
open                  float64
high                  float64
low                   float64
close                 float64
volume                float64
dtype: object
-------------------
                           date     open     high      low    close    volume
12116 2021-02-12 00:00:00+00:00  27.8645  27.9951  27.7551  27.7933  51591.83
-------------------
aberration, above, above_value, accbands, ad, adjusted, adosc, adx, amat, ao, aobv, apo, aroon, atr, bbands, below, below_value, bias, bollinger_bands, bop, brar, categories, cci, cdl_doji, cdl_inside, cfo, cg, chop, chopiness, cksp, cmf, cmo, constants, coppock, cores, cross, cross_value, crossed, crossed_above, crossed_below, datetime_ordered, decay, decreasing, dema, donchian, dpo, efi, ema, entropy, eom, er, eri, fisher, fwma, ha, heikinashi, hilo, hl2, hlc3, hma, hull_moving_average, ibs, ichimoku, implied_volatility, increasing, indicators, inertia, kama, kc, kdj, keltner_channel, kst, kurtosis, linreg, log_return, log_returns, long_run, macd, mad, massi, mcgd, median, mfi, mid_price, midpoint, midprice, mom, mp, natr, nvi, obv, ohlc4, pdist, percent_return, pgo, ppo, psar, psl, pvi, pvo, pvol, pvr, pvt, pwma, qqe, qstick, quantile, returns, reverse, rma, roc, rolling_max, rolling_mean, rolling_min, rolling_std, rolling_vwap, rolling_weighted_mean, rsi, rsx, rvgi, rvi, session, short_run, sinwma, skew, slope, sma, smi, squeeze, ssf, stdev, stoch, stochrsi, strategy, supertrend, swma, t3, tdi, tema, thermo, trend_return, trima, trix, true_range, tsi, ttm_trend, typical_price, ui, uo, variance, version, vidya, vortex, vp, vwap, vwma, wcp, weighted_bollinger_bands, willr, wma, zlema, zlhma, zlma, zlsma, zlwma, zscore

[twopirllc]

@Fredrik81,

Thanks!

Are there other indicators in the same directory as _"./freqtrade/userdata/strategies/Indicator.py" that import freqtrade.vendor.qtpylib.indicators as qtpylib? Or any file in "./freqtrade/"?

It could be stuck in the import cache. Does it still happen after clearing it?

Thanks, KJ

[Fredrik81]

@twopirllc That's it :-) I remove all other strategy files. It got past that error and now has a new error in the "dataframe.ta.strategy()".

qstick((), {'append': True})
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "./freqtrade/.env/lib/python3.8/site-packages/pandas_ta/core.py", line 433, in _mp_worker
    return getattr(self, method)(*args, **kwargs)
  File "./freqtrade/.env/lib/python3.8/site-packages/pandas_ta/core.py", line 1116, in vwap
    result = vwap(high=high, low=low, close=close, volume=volume, anchor=anchor, offset=offset, **kwargs)
  File "./freqtrade/.env/lib/python3.8/site-packages/pandas_ta/overlap/vwap.py", line 23, in vwap
    vwap  = wp.groupby(wp.index.to_period(anchor)).cumsum()
AttributeError: 'RangeIndex' object has no attribute 'to_period'
"""

But I guess I should not ask for more assistance and take more time here. Thanks a lot!

[twopirllc]

@Fredrik81,

That's it :-) I remove all other strategy files.

Excellent! Thanks for your assistance and patience with debugging why the DataFrame was getting extended prior to ta.strategy() executing.

AttributeError: 'RangeIndex' object has no attribute 'to_period'

But I guess I should not ask for more assistance and take more time here.

No worries. I recommend changing your DataFrame index to a DatatimeIndex so Pandas can anchor the period correctly. See help(ta.vwap).

import pandas as pd
import pandas_ta as ta

dataframe = # your ohlcv
dataframe.set_index(pd.DatetimeIndex(dataframe["date"]), inplace=True, drop=True)
dataframe.drop(columns="date", axis=1, inplace=True)

dataframe.ta.strategy()
print(dataframe)

Let me know if that works.

Thanks, KJ

[Fredrik81]

yeah that did the tick now it's working perfectly!

[twopirllc]

@Fredrik81,

Excellent! Glad we were able to get it to work with freqtrade. Please let me know how it performs with it.

Kind Regards, KJ

[Fredrik81]

@twopirllc Well why i'm switching is because pandas-ta seems to have better performance and feels a bit more light weight than ta-lib. What i'm working on at the moment is combining freqtrade, tensorflow and stablebaselines to see how i can get re-enforced learning to work with it. If i could wish for one thing it would be namings of the pandas columns like "[indicator][settings]". I see the point of the way it's implemented in pandas-ta but it can be acting strange in atomization etc. perhaps there is already a way to make it take away the _[settings] part in the name creation but at the moment i just do this to take out some of the characters that become a problem for me:

dataframe.columns = dataframe.columns.str.replace('[#,@,&,.]', '')

[twopirllc]

@Fredrik81,

Well why i'm switching is because pandas-ta seems to have better performance and feels a bit more light weight than ta-lib. What i'm working on at the moment is combining freqtrade, tensorflow and stable_baselines to see how i can get re-enforced learning to work with it.

Gotcha

If i could wish for one thing it would be namings of the pandas columns like "[indicator][settings]". I see the point of the way it's implemented in pandas-ta but it can be acting strange in atomization etc. perhaps there is already a way to make it take away the [settings] part in the name creation but at the moment i just do this to take out some of the characters that become a problem for me:

Yeah I understand. For the resultant column names, "[indicator]_[settings]" seemed to be the least problematic and yet descriptive way to identify them. Obviously the "." column names from indicators with float parameters makes the process less smooth in the AI/ML pipeline. I am open to suggestions for better column naming.

Yes, this works and is a simple command to adjust column names in the feature generation post-processing phase of the pipeline. I have thought about adding something similar to Pandas TA but unsure if it is needed as post-processing requirements may vary.

dataframe.columns = dataframe.columns.str.replace('[#,@,&,.]', '')

Kind Regards, KJ

[sundaraman-iyer]

Traceback (most recent call last): File "V:\Offical Docs\ML-EUR-INR\mlforex\lib\site-packages\pandas\core\indexes\base.py", line 3653, in get_loc return self._engine.get_loc(casted_key) File "pandas_libs\index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc File "pandas_libs\index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc File "pandas_libs\hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas_libs\hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'CCI_20_2.0'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "cci.py", line 14, in plt.plot(df['Date'], df['CC_20_2.0'], label='CCI', color='blue') File "V:\Offical Docs\ML-EUR-INR\mlforex\lib\site-packages\pandas\core\frame.py", line 3761, in getitem indexer = self.columns.get_loc(key) File "V:\Offical Docs\ML-EUR-INR\mlforex\lib\site-packages\pandas\core\indexes\base.py", line 3655, in get_loc raise KeyError(key) from err KeyError: 'CCI_20_2.0'

What is the column name for Commodity Channel Index in pandas_ta library?