ATR differences

[hroff-1902]

Moved this from ta-lib: https://github.com/mrjbq7/ta-lib/issues/266 (I guess we should not discuss technical and qtpylib in ta-lib repo)

The ATR indicator (which is part of DMI) implementation in qtpylib gives results that differ from all 4 variants (RMA/SMA/EMA/WMA) of tradingview's built-in ATR... Btw, ATR from talib also produces different (from all others) results for ATR. And, ATR from freqtrade/technical (which wraps ATR from pyti) also seems to be just buggy, the values are very far from all other implementations...

How to reproduce: add in the custom strategy (I'm not plotting, just printing the results):

import talib as talib
import technical.indicators as technical
...

def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        ...
        dataframe['tr'] = qtpylib.true_range(dataframe)
        dataframe['atr-q1'] = qtpylib.atr(dataframe, window=14, exp=False)
        dataframe['atr-q2'] = qtpylib.atr(dataframe, window=14, exp=True)
        dataframe['atr-technical'] = technical.atr(dataframe, period=14)
        dataframe['atr-talib'] = talib.ATR(dataframe['high'], dataframe['low'], dataframe['close'], timeperiod = 14)
        ...
        print(dataframe[['date', 'open', 'high', 'low', 'close', 'tr', 'atr-q1', 'atr-q2', 'atr-technical', 'atr-talib']])
        return dataframe

qtpylib's TR is okay, but ATR...

I used ETH/BTC, 1h from Binance (here and on tradingview)

[hroff-1902]

seems I found the bug in qtpylib's ATR, will report there. (It applies RMA over SMA/EMA smoothing, while only SMA or EMA (or RMA, right) should be used)

[hroff-1902]

posted in qtpylib: https://github.com/ranaroussi/qtpylib/pull/120

[hroff-1902]

@xmatthias notice the differencies in the values of this indicator, returned by pyti's atr, used here in technical...

[hroff-1902]

ATR in pyti is completely broken as completely broken TR (true_range). It (the TR indicator and, upper, ATR) only accepts 'close' as a parameter while calculations should be done on high, low AND close values.

technical should better move to qtpylib's TR/ATR after it will be fixed.

pyti seems for me completely inactive project for the last year,,,

[mishaker]

you sure that it is qtpylib's bug ? ATR calculation of trading view is werido

[mishaker]

this is ATR with RMA smoothing of TV:

df['TR'] = ta.TRANGE(df)
df['ATR'] = df['TR'].ewm(alpha=1 / period).mean()

[hroff-1902]

as I posted in qtpy PR, they incorrectly apply RMA smoothing AFTER they already made SMA or EMA on TR. After removal of RMA part the results (for both exp=True (EMA) and exp=False(SMA)) are identical to tradingview...

pyti's TR/ATR is completely broken in the same time, see above

[hroff-1902]

ATR is just a smoothed TR (either with EMA or SMA, or, in case of other tradingview variants, RMA and WMA) by definition, right? No reason to make EMA/SMA and then another smoothing, it's just wrong.

[mishaker]

it is plain wrong yes.

[xmatthias]

Running a few more tests with this based on the spreadsheet from here, it would appear that every calculation is wrong. (Note: the formula on wikipedia is the same another source)

Technical uses an EMA - however based on the formula in the excel that's not correct. Instead, it should use the mean as first item (14th element with timerange=14) - and then use Current ATR = [(Prior ATR x 13) + Current TR] / 14.

My implementation looks as follows and is the only one of the compared above to match the calculation from the excel, which (according to my excel knowledge and assuming Wikipedia's formula is correct) implements ATR correctly:

def calc_atr(df, high, low, close, timeperiod=14):
    df['H_L'] = df[high] - df[low]
    df['H_Cp'] = abs(df[high] - df[close].shift(1))
    df['L_Cp'] = abs(df[low] - df[close].shift(1))
    df['TR'] = df[["H_L", "H_Cp", "L_Cp"]].max(axis=1)
    df['ATR'] = df['TR'].rolling(timeperiod).mean()
    for i in range(timeperiod , len(df)):
        df.loc[i, 'ATR'] = (df.loc[i - 1, 'ATR'] * (timeperiod -1) + df.loc[i, 'TR']) / timeperiod

    return df

Test

cs_ref = pd.read_excel("cs-atr.xls", header=3)
cs_ref = cs_ref.drop(["Unnamed: 0", "Unnamed: 1"], axis=1)
cs_ref = cs_ref.rename({"TR": "TR_orig", "ATR": "ATR_orig", "High": "high", "Low": "low", "Close": "close"}, axis=1)
calc_atr(cs_towork, "high", "low", "close")

Now this is not an implementation we can use "as is" ... however it'll show a comparison of "ATR_orig" with what the different functions do - which does not seem aligned.

Compare to the ones mentioned above From other libraries:

populate_df(cs_towork)

side note:

also https://github.com/ranaroussi/qtpylib/pull/120 does not completely align the calculation, because panas.ewm is used (and i was unable to get it to compute the EWM as defined in the formula).

[mishaker]

@xmatthias ATR by definition (from wikipedia) is smoothed moving average (SMMA) of the true range values Now, all depends on your smoothing method as mentioned in above comments: RMA, SMA, EMA, WMA. I think what you are calculating is EMA (exponential moving average). But I guess you can calculate it with Pandas already?

Myself was interested in RMA smoothing so this perfectly matches the values on TV:

df['TR'] = ta.TRANGE(df)
df['ATR'] = df['TR'].ewm(alpha=1 / period).mean()

On a side note: you can easily see different values for different smoothing methods of ATR on trading view:

[xmatthias]

Wikipedia:

The first ATR value is calculated using the arithmetic mean formula:

This is exactly what i do - the only difference is the first value of the entry, as defined above via wikipedia.

I am not convinced that qtpylib rolling mean is aligned with your results:

https://github.com/hroff-1902/qtpylib/blob/d11b997ccd317c60b7de0821220add4b166dfb9f/qtpylib/indicators.py#L298-L303

[mishaker]

I am not talking about qtpylib :) I am saying that with pandas you can calculate all smoothing methods of ATR. I was only interested in RMA but I have also implemented other smoothings.

[xmatthias]

you'll however not get it 100% aligned for the first few entries - until the "initial" errors are smoothed out.

[hroff-1902]

When an year ago or some I read the Ehler's old original book, I took his algorithms written for an old lamp trading system which no one use in the meantime and reimplemented them in python... However, that system has had limitation for atan() for only 0-90 degrees. So Ehler in his original code assigned signs to the results of atan() for other 3 quorters. And they differ from those that are returned by native modern functions in python (and other libraries). That reflect significantly to the values of the phasors in his indicators. And I still do not know if it was an error in his original algorithms or intentional assignment...

What I mean -- as far as I got, people often test they algorithms first in tradingview and then move them to freqtrade strategies. We should not align to old original books written by the trading monsters many decades ago, but to the modern implementations of their algorythms existing on tradingview and other similar platforms...

[2W-12]

Somehow there is still difference with TradingView builtin ATR function. I can't match any results.

[synackSA]

I found this library which I think is better, I switched to using it and I'm getting better results: https://github.com/bukosabino/ta

[NeoDimi]

Hi there, why is this thread marked as completed when it wasn't solved? The DMI of Freqtrade and of tradingview are completely misaligned. It seems it's the DMI used by Freqtrade that's off. Probably because the ATR as described above isn't calculated right. This is a real problem because DMI and ATR are central to many strategies. Is it not fixable in ta-lib/qtpylib or freqtrade itself?

[xmatthias]

ta-lib (the underlying C library) and qtpylib is are both unmaintained as far as i know. issues in these will probably not be fixed.

for indicators provided in this library: we aim to align to the original specification, not tradingview. we've had several cases where the issue turned out to be tradingview deviating from the original paper.

Unless you can provide values / data that shows that tradingview is correct (aligned with the original paper) - and technical is not (on indicators that are NOT vendored from other libraries), there'll be little we will be doing.

also - this topic is abut ATR - not DMI.

[NeoDimi]

Ok, so you're positive the ATR/DMI used in this lib are the closest to the original definition?

The data I can provide quickly is that both trendspider and tradingview seem to use the same formula because the ATR, DI+ & DI- values on different timeframes are the same (to a certain precision, usually they don't quite match past the decimal point, which is negligeable). Whilst the data from Freqtrade is different (usually by several points).

I don't want to enter a battle of white papers here but in trading there is value to data being watched by millions of traders. An argument could be made that, should TV really deviate from the definition, a "TV_PLUS_DI" indicator would still make sense.

My issue is simply trying to port strategies designed manually with tradingview back in the day to the absolutely amazing and powerful tool that is freqtrade.

[xmatthias]

i never looked at DMI - and for ATR - we don't provide a implementation of this. It's a wrapped/vendored library from qtpylib - as we only need the indicators file from that library. That library hasn't seen an update in 16+ months (actually in 3 years - if we exclude a change from master to main branch).

You're obviously free to report any issues there anyway - but i doubt they'll be responded to.

If you aim to be aligned to tradingview, then indicators from pandas_ta might be your best bet. Their aim is to align to tradingview afaik.

[NeoDimi]

Thanks for your answer, appreciate it. I will have a look.

[zakcali]

this is a working python code values are different from tradingview or investing, but timing of +di and -di crossovers are correct

adx, +di, -di embedded, reference from: https://medium.com/codex/does-combining-adx-and-rsi-create-a-better-profitable-trading-strategy-125a90c36ac

import pandas as pd

def get_adx(high, low, close, lookback):

    plus_dm = high.diff()
    minus_dm = low.diff()
    plus_dm[plus_dm < 0] = 0
    minus_dm[minus_dm > 0] = 0
    tr1 = pd.DataFrame(high - low)
    tr2 = pd.DataFrame(abs(high - close.shift(1)))
    tr3 = pd.DataFrame(abs(low - close.shift(1)))
    frames = [tr1, tr2, tr3]
    tr = pd.concat(frames, axis = 1, join = 'inner').max(axis = 1)
    atr = tr.rolling(lookback).mean()
    plus_di = 100 * (plus_dm.ewm(alpha = 1/lookback).mean() / atr)
    minus_di = abs(100 * (minus_dm.ewm(alpha = 1/lookback).mean() / atr))
    dx = (abs(plus_di - minus_di) / abs(plus_di + minus_di)) * 100
    adx = ((dx.shift(1) * (lookback - 1)) + dx) / lookback
    adx_smooth = adx.ewm(alpha = 1/lookback).mean()
    return plus_di, minus_di, adx_smooth

df = pd.read_csv('1D/XU100-1D.csv')
tempor = get_adx (df["High"], df["Low"], df["Close"], 14)
df["PLUS_DI_14"] = tempor [0]
df["MINUS_DI_14"] = tempor [1]
df["ADX_14"] = tempor [2]
pd.set_option('display.max_rows', df.shape[0]+1)
print(df)

[acuna-public]

xmatthias original specification is ok, but there're trading platforms which ATR values are differ, so there are no guarantees that the original specification can lead to undesirable consequences, for example wrong entry signals, unwanted entry price because of earlier order creation, non-opening the orders etc. because seems that they using this values for some purposes, I don't think there is a error traversing under all platforms.