twopirllc / pandas-ta

Technical Analysis Indicators - Pandas TA is an easy to use Python 3 Pandas Extension with 150+ Indicators
https://twopirllc.github.io/pandas-ta/
MIT License
5.42k stars 1.06k forks source link

Elders Thermometer Indicator #119

Closed twopirllc closed 4 years ago

twopirllc commented 4 years ago

Which version are you running? The lastest version is on Github. Pip is for major releases.

import pandas_ta as ta
print(ta.version)

Version 0.2.01b

Upgrade.

$ pip install -U git+https://github.com/twopirllc/pandas-ta

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like Elders Thermometer Psuedocode Elders Thermometer by LazyBear MQL5 Description

Describe alternatives you've considered None at this time

Additional context None

Thanks for using Pandas TA! 😎

rluong003 commented 4 years ago

Hi, I'd like to give a shot at implementing this. Can I ask, what category of indicator is the Elders Thermometer?

twopirllc commented 4 years ago

Hello @rluong003,

Great! That is a good question because even the few sources on it do not have it explicitly categorized. It seems to measure price volatility, so let's put it in volatility. Also, let's call it thermo. You will also need to add thermo to /volatility/__init_.py. After that, I can take it from there if you like. Or you can try to do it all and I'll guide you on the other steps, just let me know. When your ready, make a Pull Request.

I also included MQL5's and LazyBear's version on TradingView on the original post for another source to compare with.

Thanks for taking this on and using Pandas TA!

Regards, KJ

rluong003 commented 4 years ago

Hi @twopirllc ,

I'd like to try to implement it myself. I need some guidance here. Looking at the implementations of Elder's Force Index, and Elder Ray Index, I sort of have an idea on how to do it.

Based off the psuedocode from motionwave, my function should parameters should be (high, low, length, mamode **kwargs) I'm not sure what offset and drift are. My question is here: prevL = low[index-1]; prevH = high[index-1];

How do I get the previous low/current low, previous high/current high?

twopirllc commented 4 years ago

@rluong003,

Cool!

Before you get too far, make sure you are using the latest version of Pandas TA. I just update the Github version earlier today.

$ pip install -U git+https://github.com/twopirllc/pandas-ta

Ok, so here is how I would start it. You can use shift() to shift a column up or down. I use a drift variable to allow shift() to be mutable by the user if needed. By default, drift=1. The mamode can be used to add some other options for a moving average if you like.

def thermo(high, low, mamode=None, drift=None, offset=None, **kwargs):
    high = verify_series(high)
    low = verify_series(low)
    mamode = mamode.lower() if mamode else "ema"
    drift = get_drift(drift)
    offset = get_offset(offset)
    # ...
    prev_high = high.shift(drift)
    prev_low  = low.shift(drift)
    # ...

Furthermore, you will need to add thermo to the volatility Category dict in pandas_ta/__init__.py. That way the ta.strategy() method can find it under volatility. As well as add the thermo method into pandas_ta/core.py alphabetically with the rest of the volatility indicators.

Finally, there are the tests to write. You will need to add them to tests/test_indicator_volatility.py and tests/test_ext_indicator_volatility.py. You do not need to worry about testing it with TA Lib because TA Lib does not have thermo.

Hope this helps.

Thanks, KJ

rluong003 commented 4 years ago

@twopirllc Thank you so much for your help, I'm new to open source contributions and investing in general, so I really appreciate your guidance. Do I also have to implement the signals? Or is that up to the user to decide after they get back their Thermo

//Signals sell = ther moreThan (ma sellFac); buy = ther lessThan (ma buyFac);

def thermo(high, low, mamode=None, drift=None, offset=None, **kwargs):
    """Indicator: Elders Thermometer (THERMO)"""
    # Validate arguments
    high = verify_series(high)
    low = verify_series(low)
    length = int(length) if length and length > 0 else 20
    mamode = mamode.lower() if mamode else "ema"
    drift = get_drift(drift)
    offset = get_offset(offset)

    # Calculate Result
    prev_high = high.shift(drift)
    prev_low  = low.shift(drift)

    thermoL = abs(prev_low - low)
    thermoH = abs(high - prev_high)

    if thermoH > thermoL:
        thermo = thermoH
    else:
        thermo = thermoL

    thermo = ema(thermo, length)

    #Offset
    if offset != 0:
        thermo = thermo.shift(offset)

    # Handle fills
    if "fillna" in kwargs:
        thermo.fillna(kwargs["fillna"], inplace=True)
    if "fill_method" in kwargs:
        thermo.fillna(method=kwargs["fill_method"], inplace=True)

    # Name and Categorize it
    thermo.name = f"THERMO_{length}"
    thermo.category = "volatility"

    return thermo
twopirllc commented 4 years ago

@rluong003,

Thank you so much for your help, I'm new to open source contributions and investing in general, so I really appreciate your guidance.

No worries. We all start somewhere. I am still learning things too. You are in a good spot if you have the time, interest, and youth to start investing early.

Do I also have to implement the signals?

Yes. But if you do not want to, I will take care of it.

Also the code looks good. You get the structure of the indicators.

You can also do something like this:

    # Calculate Result
    thermoL = (low.shift(drift) - low).abs()
    thermoH = (high - high.shift(drift)).abs()

Unless you plan to use prev_high and prev_low later for another use.

    prev_high = high.shift(drift)
    prev_low  = low.shift(drift)

Also I do a visual test by comparing it to another source, like TradingView, but use whatever you have. Since Pandas also depends on matplotlib, you can get a quick plot up in a Jupyter Notebook so you can compare with another charting platform.

Here is an example plot if it just returns a Series:

thermo = ta.thermo(df.high, df.low, length=10)
thermo.tail(252).plot(figsize=(16, 5), color=["black"], title=thermo.name, grid=True)

Doing good!

Thanks, KJ

rluong003 commented 4 years ago

@twopirllc I have added the thermo method into core.

   def thermo(self, length=None, mamode=None, drift=None, offset=None, **kwargs):
        high  = self._get_column(kwargs.pop("high", "high"))
        low   = self._get_column(kwargs.pop("low", "low"))

        result = thermo(high=high, low=low,length=length, mamode=mamode, drift=drift, offset=offset, **kwargs)
        return self._post_process(result, **kwargs)

How can I implement the signals? sell = ther moreThan (ma sellFac); buy = ther lessThan (ma buyFac);

Seems simple to implement, SellFac and buyFac have a default of .5 and 2. And we send the sell or buy signal depending on the thermo value. Where does this code go?

I'm also trying to recreate the LazyBear implementation of Elder's Thermometer on a Jupyter Notebook to compare the plots. I've downloaded some historical Bitcoin data in a csv file, and stripping the timestamp to feb 2014- oct 2014.

Finally for tests tests/test_ext_indicator_volatility.py

    def test_thermo_ext(self):
        self.data.ta.thermo(append=True)
        self.assertIsInstance(self.data, DataFrame)
        self.assertEqual(self.data.columns[-1], "THERMO_20")

tests/test_indicator_volatility.py

def test_thermo(self):
        result = pandas_ta.thermo(self.high, self.low)
        self.assertIsInstance(result, Series)
        self.assertEqual(result.name, "THERMO_20")

Is this enough for the tests? Thanks again!

rluong003 commented 4 years ago

Trying to test my thermo method in a Jupyter notebook.

I changed

    if thermoH > thermoL:
        thermo = thermoH
    else:
        thermo = thermoL

into

    thermo = np.where(thermoH > thermoL, thermoH, thermoL)

because I was getting an error " Truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all() "

This change seemed to work however, when I tried to test the thermo method I get another error.

 thermoL = (low.shift(drift) - low).abs()
    thermoH = (high - high.shift(drift)).abs()

    thermo = np.where(thermoH > thermoL, thermoH, thermoL)

    thermo = ema(thermo, length)

I get this error when I try to run it

C:\Python38\lib\site-packages\pandas_ta\volatility\thermo.py in thermo(high, low, length, mamode, drift, offset, **kwargs)
    20     thermo = np.where(thermoH > thermoL, thermoH, thermoL)
---> 21    thermo = ema(thermo, length)

C:\Python38\lib\site-packages\pandas_ta\overlap\ema.py in ema(close, length, offset, **kwargs)
   16     # Calculate Result
     17     if sma:
---> 18         close = close.copy()
     19         sma_nth = close[0:length].sum() / length
     20         close[:length - 1] = npNaN
AttributeError: 'NoneType' object has no attribute 'copy'

I'm confused, So the EMA method needs a close series, but my thermo method doesn't have a close series. What can I do?

twopirllc commented 4 years ago

@rluong003

I have added the thermo method into core.

Excellent!

Is this enough for the tests?

It's close. Since the output will be a DataFrame, as described below, you will have to modify it.

So the EMA method needs a close series, but my thermo method doesn't have a close series. What can I do?

You will have to convert it to a Series since np.where() creates a <class 'numpy.ndarray'>.

thermo = np.where(thermoH > thermoL, thermoH, thermoL)
print(type(thermo)) # => <class 'numpy.ndarray'>

# Convert/Cast to a Series while making sure
#  the new Series has the same index as either high or low.
thermo = Series(np.where(thermoH > thermoL, thermoH, thermoL), index=high.index)

However there is another way to do it without using np.where(), but that can be left as exercise on your own. Hint: It is similar to the calculation of pos and neg in trend/adx.py

Furthermore, you will have to finally return a DataFrame with columns: thermo, thermo_ma, thermo_long, thermo_short. Also thermo_long, thermo_short are binary events, so they should return as an int. You can look at momentum/macd.py or any other indicator that returns a DataFrame as an example.

Hope this helps! Keep up the good work.

Regards, KJ

rluong003 commented 4 years ago

@twopirllc I think I'm almost ready for a pull request.

I'm still a little confused about thermo_short and thermo_long. I included two new parameters, long and short, because these are supposed to be user defined buy/selling factors. sell = ther moreThan (ma sellFac); buy = ther lessThan (ma buyFac);

If I do moving average * a sell or buy factor, isn't that multiplying an int by a series?

def thermo(high, low, long=None, short=None, length=None, mamode=None, drift=None, offset=None, **kwargs):
    """Indicator: Elders Thermometer (THERMO)"""
    # Validate arguments
    high = verify_series(high)
    low = verify_series(low)
    length = int(length) if length and length > 0 else 20
    long = int(long) if long and long > 0 else 2
    short = float(short) if short and short > 0 else 0.5
    mamode = mamode.lower() if mamode else "ema"
    drift = get_drift(drift)
    offset = get_offset(offset)

    # Calculate Result
    thermoL = (low.shift(drift) - low).abs()
    thermoH = (high - high.shift(drift)).abs()

    thermo = np.where(thermoH > thermoL, thermoH, thermoL)
    thermo = Series(np.where(thermoH > thermoL, thermoH, thermoL), index=high.index)

    thermoma = ema(thermo, length)

    thermo_long = long * thermoma
    thermo_short = short * thermoma

    # Offset
    if offset != 0:
        thermo = thermo.shift(offset)

    # Handle fills
    if "fillna" in kwargs:
        thermo.fillna(kwargs["fillna"], inplace=True)
        thermoma.fillna(kwargs["fillna"], inplace=True)
    if "fill_method" in kwargs:
        thermo.fillna(method=kwargs["fill_method"], inplace=True)
        thermoma.fillna(method=kwargs["fill_method"], inplace=True)

    # Name and Categorize it
    thermo.name = f"THERMO_{length}"
    thermo_long.name = f"THERMO_{long}"
    thermo_short.name = f"THERMO_{short}"
    thermoma.name = f"THERMO_{mamode}"
    thermo.category = thermo_long.category = thermo_short.category = thermoma.category = "volatility"

    # Prepare Dataframe to return
    data = {thermo.name: thermo, thermoma.name: thermoma, thermo_long.name: thermo_long, thermo_short.name: thermo_short}
    df = DataFrame(data)
    df.name = f"THERMO_{length}"
    df.category = thermo.category

    return df

This is the data I used. 60 second data of Bitcoin between April 2014 - October 2014 2020-09-29_14-20-40

Here's the results the of thermo method, not sure what I'm looking at. There's a lot of data about 300,000 rows, so I can't really tell if this is accurate with the LazyBear implementation https://www.tradingview.com/script/HqvTuEMW-Elder-s-Market-Thermometer-LazyBear/

2020-09-29_14-21-11

twopirllc commented 4 years ago

Hey @rluong003,

Thermo Long and Short

If I do moving average * a sell or buy factor, isn't that multiplying an int/float by a series?

That's legal. Same with floats. You should able to verify within a notebook cell.

Here are some minor revisions. Double check the long/short logic!

def thermo(high, low, long=None, short=None, length=None, mamode=None, asint=True, drift=None, offset=None, **kwargs):
    # ...
    long = float(long) if long and long > 0 else 2 # It's a float also

    # Create signals
    thermo_long = thermo < thermoma * long # Returns T/F
    thermo_short = thermo > thermoma * short # Returns T/F

    # Binary output, useful for signals
    if asint:
        thermo_long = thermo_long.astype(int)  
        thermo_short = thermo_short.astype(int)

    # ...

Updating the Series and DataFrame Names

Before plotting and correlation testing you need to rename some of the resultant Series cause 3/4 of them have the same name and will overwrite each other.

    # Name and Categorize it
    _props = f"_{length}_{long}_{short}"
    thermo.name = "THERMO{_props}" # This 1st
    thermoma.name = f"THERMOma{_props}" # 2nd
    thermo_long.name = f"THERMOl{_props}" # Long 3rd
    thermo_short.name = f"THERMOs{_props}" # Short 4th

    # Prepare Dataframe to return
    # ...
    df.name = thermo.name
    # ...
    return df

Plotting and Correlation Testing

If you have a TradingView account, add LazyBear's Thermo indicator to a new basic chart. Just use ticker SPY and Daily timeframe. So the new chart should mostly be: the Candles, Volume and Volume MA, and LazyBear's Thermo Plots. Zoom out as far as you can and then you should read Issue #107 on how to export data from TradingView and do correlation testing.

That will give you a good sample to compare visually. Anywhere between 500 and 1000 rows is good. Plot the data on the last 100 or so bars to see how it looks and 300-400 bars for a bigger picture to make sure it looks consistent on a large perspective. You can test Crypto later once you are certain it looks about right or its nearly 99% correlated.

So you are not plotting all the data, use tail().

recent = 100
thermo = ta.thermo(df['High'], df['Low'], length=22)
thermo = thermo.tail(recent)

# Thermo returns bars, column 0
thermo[thermo.columns[0]].plot(figsize=(16, 4), kind='bar', stacked=True, color=["black"], title=thermo.name, grid=True)

# ThermoMA is usually a line, column 1
thermo[thermo.columns[1]].plot(figsize=(16, 4), color=["black"], title=thermo.name, grid=True)

Almost there!

Thanks, KJ

rluong003 commented 4 years ago

@twopirllc Made the necessary revisions.


    thermo = Series(np.where(thermoH > thermoL, thermoH, thermoL), index=high.index)

    thermoma = ema(thermo, length)

    # Create signals
    thermo_long = thermo < (thermoma * long) # Returns T/F
    thermo_short = thermo > (thermoma * short) # Returns T/F

    # Binary output, useful for signals

    thermo_long = thermo_long.astype(int)  
    thermo_short = thermo_short.astype(int)

    # Offset
    if offset != 0:
        thermo = thermo.shift(offset)

    # Handle fills
    if "fillna" in kwargs:
        thermo.fillna(kwargs["fillna"], inplace=True)
        thermoma.fillna(kwargs["fillna"], inplace=True)
    if "fill_method" in kwargs:
        thermo.fillna(method=kwargs["fill_method"], inplace=True)
        thermoma.fillna(method=kwargs["fill_method"], inplace=True)

    # Name and Categorize it
    _props = f"_{length}_{long}_{short}"
    thermo.name = f"THERMO{_props}"
    thermoma.name = f"THERMOma{_props}"
    thermo_long.name = f"THERMOl{_props}"
    thermo_short.name = f"THERMOs{_props}"

However I don't think I'm getting a good percentage of correlation. The graphs look close, but not close enough for 99% correlation.

SPY daily data taken from TraderView with lazy bear's script

           Volume MA  Market Thermometer  EMA of Market Thermometer  
2479  65892087.05              3.6600                   0.959635  
2480  67513913.85              1.4400                   1.001406  
2481  69390477.30              0.7200                   0.976936  
2482  70580901.80              0.6400                   0.947637  
2483  78328172.40              8.8900                   1.638277  
...           ...                 ...                        ...  
2974  83534229.85              3.7600                   3.366494  
2975  84828784.60              2.3000                   3.273756  
2976  85480548.80              2.7830                   3.231081  
2977  86280332.50              5.3800                   3.417944  
2978  85549038.65              0.5291                   3.166740  

My thermo data

          THERMO_22_3.0_7.0  THERMOma_22_3.0_7.0  THERMOl_22_3.0_7.0    THERMOs_22_3.0_7.0
2479             3.6600             1.232652                   1                           0
2480             1.4400             1.250683                   1                           0
2481             2.0500             1.320188                   1                           0
2482             1.2700             1.315824                   1                           0
2483             8.8900             1.974448                   0                           0
...                 ...                  ...                 ...   
2974             3.7600             4.229678                   1                           0
2975             4.4030             4.244750                   1                           0
2976             2.7830             4.117641                   1                           0
2977            10.5100             4.673498                   1                           0
2978             0.5291             4.313116                   1                           0

The moving averages are nowhere close to each other. Any thoughts?

twopirllc commented 4 years ago

@rluong003

However I don't think I'm getting a good percentage of correlation.

Well "THERMO_22_3.0_7.0" "looks" like it is working out. But you have yet to show numerical value of correlation. More rigor is needed beyond printing two separate DataFrames.

To simplify the correlation output, combine the last two columns of the first DataFrame and the first two columns of the second DataFrame into one DataFrame say results.

results = # combined df
print(results.columns) # should match 'in name': ['Market Thermometer', 'EMA of Market Thermometer', 'THERMO_22_3.0_7.0', 'THERMOma_22_3.0_7.0']

results.corr() # Check their correlation


The moving averages are nowhere close to each other. Any thoughts?

🤷‍♂️ I'll check tomorrow.

KJ

rluong003 commented 4 years ago

@twopirllc Oh wow, 90% correlation on my therm, and 99% correlation on my thermo MA. Is 90% correlation acceptable?

                                    Market Thermometer  EMA of Market Thermometer  \
Market Thermometer                   1.000000                   0.587258   
EMA of Market Thermometer            0.587258                   1.000000   
THERMO_22_3.0_7.0                    0.904005                   0.671547   
THERMOma_22_3.0_7.0                  0.551746                   0.993868  
twopirllc commented 4 years ago

@rluong003

Thanks.

That is weird that your THEMRO is off but THERMma is accurate. It might be the case that your THERMO is also correct since Lazy Bear has this thermo = iff (high<high[1] and low>low[1], 0, ...) condition which sets THERMO to zero if the current high is less than the previous high and the current low is greater than the previous low. You can include a Lazy Bear version into the indicator if you want. For an example, check out momentum/squeeze.py for the logic.

Also, instead of using np.where() you can use the Pandas Series where so you do not have to cast the np.where() to Series.

It's looking better.

KJ

rluong003 commented 4 years ago

@twopirllc

While implementing the Lazy Bear version, I feel like its using the exact same calculations I use.

Still getting 90% THERMO correlation after using the Pandas where() instead of the np.where() Not sure how to proceed. May I submit a PR?

thermo = thermoL
thermo = thermo.where(thermoH < thermoL, thermoH)

Also I've modified the test, which passed after I ran the unit tests

    def test_thermo_ext(self):
        self.data.ta.thermo(append=True)
        self.assertIsInstance(self.data, DataFrame)
        self.assertEqual(list(self.data.columns[-4:]), ["THERMO_20_2_0.5", "THERMOma_20_2_0.5", "THERMOl_20_2_0.5", "THERMOs_20_2_0.5"])

Since talib has no thermo , should i still include this test in test_indicator_volatility?

  def test_thermo(self):
        result = pandas_ta.thermo(self.high, self.low)
        self.assertIsInstance(result, DataFrame)
        self.assertEqual(result.name, "THERMO_20_2_0.5")
twopirllc commented 4 years ago

@rluong003,

May I submit a PR?

Yes. It is fine. I typically play around with it. Getting to 90% is good for a non-traditional TA Lib indicator. Sometimes it's impossible to hit 99% or more. In all honesty, I spend considerable time to make the indicators as accurate as possible. I would rather they be accurate before they are fast, despite others preferring the latter. I do not always get 99%+ correlation with indicators initially.

Also I've modified the test, which passed after I ran the unit tests.

Perfect!

Since talib has no thermo , should i still include this test in test_indicator_volatility?

Yes. We want to ensure that a DataFrame with the proper name is returned.

Overall, good job! Perhaps you may want to try something a bit more involved like Issue #85, the QQE indicator or #120 Bollinger Bands Bandwith? Also there are some other TA Lib indicators on the Projects Board so you can see how you do vs. TA Lib if you are up to the challenge.

After all, practice makes you better. There is no rush, but eventually I have to get it completed. It would be a great help to have someone to be able to at a minimum get the majority of the scaffolding done while I make progress in other worthwhile features in mind.

Thank you for the contribution! KJ