WARNING - Unable to analyze candle (OHLCV) data for pair ETH/USDT: Dataframe returned from strategy has mismatching last close price.

[Knightmare6890]

I am using a customized GodStra by Mablue with ta. I am trying to check previous candles for a trend in "populate_indicators", but I suspect that somehow the index referencing is not returning the dataframe correctly, and in dry run it gives me above warning. In backtest and hyperopt it works fine.

Please see code below. If anyone has an idea how to fix this warning, please let me know.

# GodStraNew Strategy
# Author: @Mablue (Masoud Azizi)
# github: https://github.com/mablue/
# freqtrade hyperopt --hyperopt-loss SharpeHyperOptLoss --spaces buy roi trailing sell --strategy GodStraNew
# --- Do not remove these libs ---
from freqtrade import data
from freqtrade.strategy.hyper import CategoricalParameter, DecimalParameter

from numpy.lib import extract, math
from freqtrade.strategy.interface import IStrategy
from pandas import DataFrame

# --------------------------------

# Add your lib to import here
# TODO: talib is fast but have not more indicators
# import talib.abstract as ta
import ta
import freqtrade.vendor.qtpylib.indicators as qtpylib
from functools import reduce
import numpy as np
from random import shuffle
import inspect

all_god_genes = {
    'ohlcv': {
        'open', 'high', 'low', 'close', 'volume'
    },
    'volume': {
        'volume_adi',
        'volume_obv', 'volume_cmf', 'volume_fi', 'volume_mfi', 'volume_em',
        'volume_sma_em', 'volume_vpt', 'volume_nvi', 'volume_vwap',
    },
    'volatility': {
        'volatility_atr', 'volatility_bbm', 'volatility_bbh', 'volatility_bbl',
        'volatility_bbw', 'volatility_bbp', 'volatility_bbhi',
        'volatility_bbli', 'volatility_kcc', 'volatility_kch', 'volatility_kcl',
        'volatility_kcw', 'volatility_kcp', 'volatility_kchi',
        'volatility_kcli', 'volatility_dcl', 'volatility_dch', 'volatility_dcm',
        'volatility_dcw', 'volatility_dcp', 'volatility_ui',
    },
    'trend': {
        'trend_macd',
        'trend_macd_signal', 'trend_macd_diff', 'trend_sma_fast',
        'trend_sma_slow', 'trend_ema_fast', 'trend_ema_slow', 'trend_adx',
        'trend_adx_pos', 'trend_adx_neg', 'trend_vortex_ind_pos',
        'trend_vortex_ind_neg', 'trend_vortex_ind_diff', 'trend_trix',
        'trend_mass_index', 'trend_cci', 'trend_dpo', 'trend_kst',
        'trend_kst_sig', 'trend_kst_diff', 'trend_ichimoku_conv',
        'trend_ichimoku_base', 'trend_ichimoku_a', 'trend_ichimoku_b',
        'trend_visual_ichimoku_a', 'trend_visual_ichimoku_b', 'trend_aroon_up',
        'trend_aroon_down', 'trend_aroon_ind', 'trend_psar_up',
        'trend_psar_down', 'trend_psar_up_indicator',
        'trend_psar_down_indicator', 'trend_stc',
    },
    'momentum': {
        'momentum_rsi',
        'momentum_stoch_rsi', 'momentum_stoch_rsi_k', 'momentum_stoch_rsi_d',
        'momentum_tsi', 'momentum_uo', 'momentum_stoch',
        'momentum_stoch_signal', 'momentum_wr', 'momentum_ao', 'momentum_kama',
        'momentum_roc', 'momentum_ppo', 'momentum_ppo_signal',
        'momentum_ppo_hist',
    },
    'other': {
        'others_dr', 'others_dlr', 'others_cr'
    }
}
print(all_god_genes)

god_genes = set()
########################### SETTINGS ##############################
god_genes = {'trend_adx', 'volatility_bbm', 'volatility_bbh', 'volatility_bbl', 
        'volume_obv', 'momentum_stoch', 'momentum_stoch_signal'}
# god_genes |= all_god_genes['ohlcv']
# god_genes |= all_god_genes['volume']
# god_genes |= all_god_genes['volatility']
# god_genes |= all_god_genes['trend']
# god_genes |= all_god_genes['momentum']
# god_genes |= all_god_genes['other']
operators = [
    "D",  # Disabled gene
    ">",  # Indicator, bigger than cross indicator
    "<",  # Indicator, smaller than cross indicator
    "=",  # Indicator, equal with cross indicator
    "C",  # Indicator, crossed the cross indicator
    "CA",  # Indicator, crossed above the cross indicator
    "CB",  # Indicator, crossed below the cross indicator
    ">R",  # Normalized indicator, bigger than real number
    "=R",  # Normalized indicator, equal with real number
    "<R",  # Normalized indicator, smaller than real number
    "/>R",  # Normalized indicator devided to cross indicator, bigger than real number
    "/=R",  # Normalized indicator devided to cross indicator, equal with real number
    "/<R",  # Normalized indicator devided to cross indicator, smaller than real number
    "UT",  # Indicator, is in UpTrend status
    "DT",  # Indicator, is in DownTrend status
    "OT",  # Indicator, is in Off trend status(RANGE)
    "CUT",  # Indicator, Entered to UpTrend status
    "CDT",  # Indicator, Entered to DownTrend status
    "COT"  # Indicator, Entered to Off trend status(RANGE)
]
# number of candles to check up,don,off trend.
TREND_CHECK_CANDLES = 14
DECIMALS = 1
########################### END SETTINGS ##########################

# Let give somethings to CatagoricalParam to Play with them
# When just one thing is inside catagorical lists
# TODO: its Not True Way :)
if len(god_genes) == 1:
    god_genes = god_genes*2
if len(operators) == 1:
    operators = operators*2

def normalize(df):
    df = (df-df.min())/(df.max()-df.min())
    return df

def condition_generator(dataframe, operator, indicator, crossed_indicator, real_num):

    condition = (dataframe['volume'] > 10)

    # TODO : it ill callculated in populate indicators.

    indicator_trend_sma = f"{indicator}-SMA-{TREND_CHECK_CANDLES}"

    if operator == ">":
        condition = (
            dataframe[indicator] > dataframe[crossed_indicator]
        )
    elif operator == "=":
        condition = (
            np.isclose(dataframe[indicator], dataframe[crossed_indicator])
        )
    elif operator == "<":
        condition = (
            dataframe[indicator] < dataframe[crossed_indicator]
        )
    elif operator == "C":
        condition = (
            (qtpylib.crossed_below(dataframe[indicator], dataframe[crossed_indicator])) |
            (qtpylib.crossed_above(dataframe[indicator], dataframe[crossed_indicator]))
        )
    elif operator == "CA":
        condition = (
            qtpylib.crossed_above(dataframe[indicator], dataframe[crossed_indicator])
        )
    elif operator == "CB":
        condition = (
            qtpylib.crossed_below(
                dataframe[indicator], dataframe[crossed_indicator])
        )
    elif operator == ">R":
        condition = (
            dataframe[indicator] > real_num
        )
    elif operator == "=R":
        condition = (
            np.isclose(dataframe[indicator], real_num)
        )
    elif operator == "<R":
        condition = (
            dataframe[indicator] < real_num
        )
    elif operator == "/>R":
        condition = (
            dataframe[indicator].div(dataframe[crossed_indicator]) > real_num
        )
    elif operator == "/=R":
        condition = (
            np.isclose(dataframe[indicator].div(dataframe[crossed_indicator]), real_num)
        )
    elif operator == "/<R":
        condition = (
            dataframe[indicator].div(dataframe[crossed_indicator]) < real_num
        )
    elif operator == "UT":
        condition = (
            dataframe[indicator] > dataframe[indicator_trend_sma]
        )
    elif operator == "DT":
        condition = (
            dataframe[indicator] < dataframe[indicator_trend_sma]
        )
    elif operator == "OT":
        condition = (

            np.isclose(dataframe[indicator], dataframe[indicator_trend_sma])
        )
    elif operator == "CUT":
        condition = (
            (
                qtpylib.crossed_above(
                    dataframe[indicator],
                    dataframe[indicator_trend_sma]
                )
            ) &
            (
                dataframe[indicator] > dataframe[indicator_trend_sma]
            )
        )
    elif operator == "CDT":
        condition = (
            (
                qtpylib.crossed_below(
                    dataframe[indicator],
                    dataframe[indicator_trend_sma]
                )
            ) &
            (
                dataframe[indicator] < dataframe[indicator_trend_sma]
            )
        )
    elif operator == "COT":
        condition = (
            (
                (
                    qtpylib.crossed_below(
                        dataframe[indicator],
                        dataframe[indicator_trend_sma]
                    )
                ) |
                (
                    qtpylib.crossed_above(
                        dataframe[indicator],
                        dataframe[indicator_trend_sma]
                    )
                )
            ) &
            (
                np.isclose(
                    dataframe[indicator],
                    dataframe[indicator_trend_sma]
                )
            )
        )

    return condition, dataframe

class GodStraNewTa(IStrategy):
    # #################### RESULTS PASTE PLACE ####################
    # *    8/24:     27 trades. 6/0/21 Wins/Draws/Losses. Avg profit  151.06%. Median profit -10.18%. Total profit  0.40827666 BTC ( 408.28Σ%). Avg duration 2 days, 5:12:00 min. Objective: -7.13571

    # Buy hyperspace params:
    buy_params = {
        "buy_crossed_indicator0": "trend_adx",
        "buy_crossed_indicator1": "volatility_bbm",
        "buy_crossed_indicator2": "momentum_stoch_signal",
        "buy_crossed_indicator3": "volatility_bbm",
        "buy_indicator0": "momentum_stoch_signal",
        "buy_indicator1": "volatility_bbh",
        "buy_indicator2": "momentum_stoch",
        "buy_indicator3": "momentum_stoch",
        "buy_operator0": "<",
        "buy_operator1": ">",
        "buy_operator2": "<",
        "buy_operator3": "/>R",
        "buy_real_num0": 0.4,
        "buy_real_num1": 0.6,
        "buy_real_num2": 0.9,
        "buy_real_num3": 0.3,
    }

    # Sell hyperspace params:
    sell_params = {
        "sell_crossed_indicator0": "momentum_stoch_signal",
        "sell_crossed_indicator1": "volatility_bbm",
        "sell_indicator0": "volatility_bbh",
        "sell_indicator1": "volatility_bbl",
        "sell_operator0": "CDT",
        "sell_operator1": "UT",
        "sell_real_num0": 1.0,
        "sell_real_num1": 0.5,
    }

    # Stoploss:
    stoploss = -1

    # #################### END OF RESULT PLACE ####################

    # TODO: Its not dry code!
    # Buy Hyperoptable Parameters/Spaces.

    buy_crossed_indicator0 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='buy')
    buy_crossed_indicator1 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='buy')
    buy_crossed_indicator2 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='buy')
    buy_crossed_indicator3 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='buy')

    buy_indicator0 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='buy')
    buy_indicator1 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='buy')
    buy_indicator2 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='buy')
    buy_indicator3 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='buy')

    buy_operator0 = CategoricalParameter(operators, default="/<R", space='buy')
    buy_operator1 = CategoricalParameter(operators, default="<R", space='buy')
    buy_operator2 = CategoricalParameter(operators, default="CB", space='buy')
    buy_operator3 = CategoricalParameter(operators, default="<R", space='buy')

    buy_real_num0 = DecimalParameter(0, 1, decimals=DECIMALS,  default=0.89009, space='buy')
    buy_real_num1 = DecimalParameter(0, 1, decimals=DECIMALS, default=0.56953, space='buy')
    buy_real_num2 = DecimalParameter(0, 1, decimals=DECIMALS, default=0.38365, space='buy')
    buy_real_num3 = DecimalParameter(0, 1, decimals=DECIMALS, default=0.56953, space='buy')

    # Sell Hyperoptable Parameters/Spaces.

    sell_crossed_indicator0 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='sell')
    sell_crossed_indicator1 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='sell')

    sell_indicator0 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='sell')
    sell_indicator1 = CategoricalParameter(
        god_genes, default="momentum_rsi", space='sell')

    sell_operator0 = CategoricalParameter(operators, default="<R", space='sell')
    sell_operator1 = CategoricalParameter(operators, default="D", space='sell')

    sell_real_num0 = DecimalParameter(0, 1, decimals=DECIMALS, default=0.09731, space='sell')
    sell_real_num1 = DecimalParameter(0, 1, decimals=DECIMALS, default=0.81657, space='sell')

    # Buy hypers
    timeframe = '15m'

    def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
        from ta import add_all_ta_features
        from ta.utils import dropna

        dataframe = dropna(dataframe)

        # Add ta features filling NaN values

        dataframe = add_all_ta_features(
            dataframe, open="open", high="high", low="low", close="close", volume="volume", fillna=True)

        for key in dataframe.keys()[1:]:
            indicator_trend_sma = f"{key}-SMA-{TREND_CHECK_CANDLES}"
            dataframe[indicator_trend_sma] = ta.trend.sma_indicator(
                dataframe[key], window=TREND_CHECK_CANDLES)

        for key in dataframe.keys()[1:]:
            dataframe[key] = normalize(dataframe[key])
        print(" ", metadata['pair'], "                 ", end="\r")

        return dataframe

    def populate_buy_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:

        conditions = list()

        # TODO: Its not dry code!
        buy_indicator = self.buy_indicator0.value
        buy_crossed_indicator = self.buy_crossed_indicator0.value
        buy_operator = self.buy_operator0.value
        buy_real_num = self.buy_real_num0.value
        condition, dataframe = condition_generator(
            dataframe,
            buy_operator,
            buy_indicator,
            buy_crossed_indicator,
            buy_real_num
        )
        conditions.append(condition)
        # backup
        buy_indicator = self.buy_indicator1.value
        buy_crossed_indicator = self.buy_crossed_indicator1.value
        buy_operator = self.buy_operator1.value
        buy_real_num = self.buy_real_num1.value

        condition, dataframe = condition_generator(
            dataframe,
            buy_operator,
            buy_indicator,
            buy_crossed_indicator,
            buy_real_num
        )
        conditions.append(condition)

        buy_indicator = self.buy_indicator2.value
        buy_crossed_indicator = self.buy_crossed_indicator2.value
        buy_operator = self.buy_operator2.value
        buy_real_num = self.buy_real_num2.value
        condition, dataframe = condition_generator(
            dataframe,
            buy_operator,
            buy_indicator,
            buy_crossed_indicator,
            buy_real_num
        )
        conditions.append(condition)

        buy_indicator = self.buy_indicator3.value
        buy_crossed_indicator = self.buy_crossed_indicator3.value
        buy_operator = self.buy_operator3.value
        buy_real_num = self.buy_real_num3.value
        condition, dataframe = condition_generator(
            dataframe,
            buy_operator,
            buy_indicator,
            buy_crossed_indicator,
            buy_real_num
        )
        conditions.append(condition)

        if conditions:
            dataframe.loc[
                reduce(lambda x, y: x & y, conditions),
                'sell']=0
        if conditions:
            dataframe.loc[
                reduce(lambda x, y: x & y, conditions),
                'buy']=1

        return dataframe

    def populate_sell_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:

        conditions = list()
        # TODO: Its not dry code!
        sell_indicator = self.sell_indicator0.value
        sell_crossed_indicator = self.sell_crossed_indicator0.value
        sell_operator = self.sell_operator0.value
        sell_real_num = self.sell_real_num0.value
        condition, dataframe = condition_generator(
            dataframe,
            sell_operator,
            sell_indicator,
            sell_crossed_indicator,
            sell_real_num
        )
        conditions.append(condition)

        sell_indicator = self.sell_indicator1.value
        sell_crossed_indicator = self.sell_crossed_indicator1.value
        sell_operator = self.sell_operator1.value
        sell_real_num = self.sell_real_num1.value
        condition, dataframe = condition_generator(
            dataframe,
            sell_operator,
            sell_indicator,
            sell_crossed_indicator,
            sell_real_num
        )
        conditions.append(condition)

        if conditions:
            dataframe.loc[
                reduce(lambda x, y: x & y, conditions),
                'buy']=0
        if conditions:
            dataframe.loc[
                reduce(lambda x, y: x & y, conditions),
                'sell']=1

        return dataframe

[xmatthias]

On a quick glance i'd say it's this section:

        for key in dataframe.keys()[1:]:
            dataframe[key] = normalize(dataframe[key])

On a glance, it seems to normalize ALL columns (including the ohlcv columns). By doing this, it'll raise this warning in live/dry. Worse - it'll also make backtesting unreliable / invalid - and will now use these "modified" prices.

[mablue]

ops yes thanks @xmatthias @Knightmare6890 change this [1:]to [6:] or [5:] to fix it

[Knightmare6890]

Thank you Matthias and mablue!

Changing the index to [5:] worked, it's now running normal.

[Chyang777]

Hello @xmatthias and @mablue, thanks a lot for your sharing on this project. Actually I'm new on python and coding too, but I have great interest learning it. I currently try to get an understanding of this GodStraNew strategy workflow, but many things I still understand. Could you please describe the meaning of this lines : god_genes_with_timeperiod = list() for god_gene in god_genes: for timeperiod in timeperiods: god_genes_with_timeperiod.append(f'{god_gene}-{timeperiod}')

Thank you very much.

[mablue]

Hello @xmatthias and @mablue, thanks a lot for your sharing on this project. Actually I'm new on python and coding too, but I have great interest learning it. I currently try to get an understanding of this GodStraNew strategy workflow, but many things I still understand. Could you please describe the meaning of this lines : god_genes_with_timeperiod = list() for god_gene in god_genes: for timeperiod in timeperiods: god_genes_with_timeperiod.append(f'{god_gene}-{timeperiod}')

Thank you very much.

this lines just make a list of strings that will be add to main dataframe. a list that have gene names with splited timeframes. for example if you setted timeframes to 7 and 14 and have a SMA gene than this lines append two strings to the list: SMA-7 and SMA-14 than we check all strings and findout which gene will calculate in which tf.and append it with specific name to the main dataframe

[Chyang777]

Thanks for your comment @mablue I will try to figure it out