Closed italodamato closed 1 year ago
xmatthias
commented
2 years ago we'll definitely need some code you're using (what the parameters are you call that function with).
Also - what's the result if you run backtesting on the same strategy/code?
italodamato
commented
2 years ago I'm investigating further. I've found that the pair where the error happens is LUNA/USDT on Binance. I wonder if the huge crash is causing this. I'm not why this happens only on with --analyze-per-epoch enabled though.
Backtesting and normal hyperopt works fine
italodamato
commented
2 years ago I'm not sure why but the error doesn't happen if you call qtpylib before pandas_ta.cti - do you have any ideas? You can also write it like this and it fixes it:
def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
qtpylib
pandas_ta.cti(dataframe.close, 100)well not having seen your original call - no?
italodamato
commented
2 years ago you can literally use the SampleStrategy file below. If you comment out import freqtrade.vendor.qtpylib.indicators as qtpylib, you will get the warning on LUNA 4h.
# pragma pylint: disable=missing-docstring, invalid-name, pointless-string-statement
# flake8: noqa: F401
# isort: skip_file
# --- Do not remove these libs ---
import numpy as np # noqa
import pandas as pd # noqa
from pandas import DataFrame
from freqtrade.strategy import (BooleanParameter, CategoricalParameter, DecimalParameter,
IStrategy, IntParameter)
# --------------------------------
# Add your lib to import here
import talib.abstract as ta
import freqtrade.vendor.qtpylib.indicators as qtpylib
import pandas_ta as pta
# This class is a sample. Feel free to customize it.
class SampleStrategy(IStrategy):
"""
This is a sample strategy to inspire you.
More information in https://www.freqtrade.io/en/latest/strategy-customization/
You can:
:return: a Dataframe with all mandatory indicators for the strategies
- Rename the class name (Do not forget to update class_name)
- Add any methods you want to build your strategy
- Add any lib you need to build your strategy
You must keep:
- the lib in the section "Do not remove these libs"
- the methods: populate_indicators, populate_entry_trend, populate_exit_trend
You should keep:
- timeframe, minimal_roi, stoploss, trailing_*
"""
# Strategy interface version - allow new iterations of the strategy interface.
# Check the documentation or the Sample strategy to get the latest version.
INTERFACE_VERSION = 3
# Can this strategy go short?
can_short: bool = False
# Minimal ROI designed for the strategy.
# This attribute will be overridden if the config file contains "minimal_roi".
minimal_roi = {
"60": 0.01,
"30": 0.02,
"0": 0.04
}
# Optimal stoploss designed for the strategy.
# This attribute will be overridden if the config file contains "stoploss".
stoploss = -0.10
# Trailing stoploss
trailing_stop = False
# trailing_only_offset_is_reached = False
# trailing_stop_positive = 0.01
# trailing_stop_positive_offset = 0.0 # Disabled / not configured
# Optimal timeframe for the strategy.
timeframe = '4h'
# Run "populate_indicators()" only for new candle.
process_only_new_candles = False
# These values can be overridden in the config.
use_exit_signal = True
exit_profit_only = False
ignore_roi_if_entry_signal = False
# Hyperoptable parameters
buy_rsi = IntParameter(low=1, high=50, default=30, space='buy', optimize=True, load=True)
sell_rsi = IntParameter(low=50, high=100, default=70, space='sell', optimize=True, load=True)
short_rsi = IntParameter(low=51, high=100, default=70, space='sell', optimize=True, load=True)
exit_short_rsi = IntParameter(low=1, high=50, default=30, space='buy', optimize=True, load=True)
# Number of candles the strategy requires before producing valid signals
startup_candle_count: int = 30
# Optional order type mapping.
order_types = {
'entry': 'limit',
'exit': 'limit',
'stoploss': 'market',
'stoploss_on_exchange': False
}
# Optional order time in force.
order_time_in_force = {
'entry': 'gtc',
'exit': 'gtc'
}
plot_config = {
'main_plot': {
'tema': {},
'sar': {'color': 'white'},
},
'subplots': {
"MACD": {
'macd': {'color': 'blue'},
'macdsignal': {'color': 'orange'},
},
"RSI": {
'rsi': {'color': 'red'},
}
}
}
def informative_pairs(self):
"""
Define additional, informative pair/interval combinations to be cached from the exchange.
These pair/interval combinations are non-tradeable, unless they are part
of the whitelist as well.
For more information, please consult the documentation
:return: List of tuples in the format (pair, interval)
Sample: return [("ETH/USDT", "5m"),
("BTC/USDT", "15m"),
]
"""
return []
def populate_indicators(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
"""
Adds several different TA indicators to the given DataFrame
Performance Note: For the best performance be frugal on the number of indicators
you are using. Let uncomment only the indicator you are using in your strategies
or your hyperopt configuration, otherwise you will waste your memory and CPU usage.
:param dataframe: Dataframe with data from the exchange
:param metadata: Additional information, like the currently traded pair
:return: a Dataframe with all mandatory indicators for the strategies
"""
# Momentum Indicators
# ------------------------------------
# ADX
dataframe['adx'] = ta.ADX(dataframe)
# # Plus Directional Indicator / Movement
# dataframe['plus_dm'] = ta.PLUS_DM(dataframe)
# dataframe['plus_di'] = ta.PLUS_DI(dataframe)
# # Minus Directional Indicator / Movement
# dataframe['minus_dm'] = ta.MINUS_DM(dataframe)
# dataframe['minus_di'] = ta.MINUS_DI(dataframe)
# # Aroon, Aroon Oscillator
# aroon = ta.AROON(dataframe)
# dataframe['aroonup'] = aroon['aroonup']
# dataframe['aroondown'] = aroon['aroondown']
# dataframe['aroonosc'] = ta.AROONOSC(dataframe)
# # Awesome Oscillator
# dataframe['ao'] = qtpylib.awesome_oscillator(dataframe)
# # Keltner Channel
# keltner = qtpylib.keltner_channel(dataframe)
# dataframe["kc_upperband"] = keltner["upper"]
# dataframe["kc_lowerband"] = keltner["lower"]
# dataframe["kc_middleband"] = keltner["mid"]
# dataframe["kc_percent"] = (
# (dataframe["close"] - dataframe["kc_lowerband"]) /
# (dataframe["kc_upperband"] - dataframe["kc_lowerband"])
# )
# dataframe["kc_width"] = (
# (dataframe["kc_upperband"] - dataframe["kc_lowerband"]) / dataframe["kc_middleband"]
# )
# # Ultimate Oscillator
# dataframe['uo'] = ta.ULTOSC(dataframe)
# # Commodity Channel Index: values [Oversold:-100, Overbought:100]
# dataframe['cci'] = ta.CCI(dataframe)
# RSI
dataframe['rsi'] = ta.RSI(dataframe)
# # Inverse Fisher transform on RSI: values [-1.0, 1.0] (https://goo.gl/2JGGoy)
# rsi = 0.1 * (dataframe['rsi'] - 50)
# dataframe['fisher_rsi'] = (np.exp(2 * rsi) - 1) / (np.exp(2 * rsi) + 1)
# # Inverse Fisher transform on RSI normalized: values [0.0, 100.0] (https://goo.gl/2JGGoy)
# dataframe['fisher_rsi_norma'] = 50 * (dataframe['fisher_rsi'] + 1)
# # Stochastic Slow
# stoch = ta.STOCH(dataframe)
# dataframe['slowd'] = stoch['slowd']
# dataframe['slowk'] = stoch['slowk']
# Stochastic Fast
stoch_fast = ta.STOCHF(dataframe)
dataframe['fastd'] = stoch_fast['fastd']
dataframe['fastk'] = stoch_fast['fastk']
# # Stochastic RSI
# Please read https://github.com/freqtrade/freqtrade/issues/2961 before using this.
# STOCHRSI is NOT aligned with tradingview, which may result in non-expected results.
# stoch_rsi = ta.STOCHRSI(dataframe)
# dataframe['fastd_rsi'] = stoch_rsi['fastd']
# dataframe['fastk_rsi'] = stoch_rsi['fastk']
# MACD
macd = ta.MACD(dataframe)
dataframe['macd'] = macd['macd']
dataframe['macdsignal'] = macd['macdsignal']
dataframe['macdhist'] = macd['macdhist']
# MFI
dataframe['mfi'] = ta.MFI(dataframe)
# # ROC
# dataframe['roc'] = ta.ROC(dataframe)
# Overlap Studies
# ------------------------------------
# Bollinger Bands
# bollinger = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=2)
# dataframe['bb_lowerband'] = bollinger['lower']
# dataframe['bb_middleband'] = bollinger['mid']
# dataframe['bb_upperband'] = bollinger['upper']
# dataframe["bb_percent"] = (
# (dataframe["close"] - dataframe["bb_lowerband"]) /
# (dataframe["bb_upperband"] - dataframe["bb_lowerband"])
# )
# dataframe["bb_width"] = (
# (dataframe["bb_upperband"] - dataframe["bb_lowerband"]) / dataframe["bb_middleband"]
# )
# Bollinger Bands - Weighted (EMA based instead of SMA)
# weighted_bollinger = qtpylib.weighted_bollinger_bands(
# qtpylib.typical_price(dataframe), window=20, stds=2
# )
# dataframe["wbb_upperband"] = weighted_bollinger["upper"]
# dataframe["wbb_lowerband"] = weighted_bollinger["lower"]
# dataframe["wbb_middleband"] = weighted_bollinger["mid"]
# dataframe["wbb_percent"] = (
# (dataframe["close"] - dataframe["wbb_lowerband"]) /
# (dataframe["wbb_upperband"] - dataframe["wbb_lowerband"])
# )
# dataframe["wbb_width"] = (
# (dataframe["wbb_upperband"] - dataframe["wbb_lowerband"]) /
# dataframe["wbb_middleband"]
# )
# # EMA - Exponential Moving Average
# dataframe['ema3'] = ta.EMA(dataframe, timeperiod=3)
# dataframe['ema5'] = ta.EMA(dataframe, timeperiod=5)
# dataframe['ema10'] = ta.EMA(dataframe, timeperiod=10)
# dataframe['ema21'] = ta.EMA(dataframe, timeperiod=21)
# dataframe['ema50'] = ta.EMA(dataframe, timeperiod=50)
# dataframe['ema100'] = ta.EMA(dataframe, timeperiod=100)
# # SMA - Simple Moving Average
# dataframe['sma3'] = ta.SMA(dataframe, timeperiod=3)
# dataframe['sma5'] = ta.SMA(dataframe, timeperiod=5)
# dataframe['sma10'] = ta.SMA(dataframe, timeperiod=10)
# dataframe['sma21'] = ta.SMA(dataframe, timeperiod=21)
# dataframe['sma50'] = ta.SMA(dataframe, timeperiod=50)
# dataframe['sma100'] = ta.SMA(dataframe, timeperiod=100)
# Parabolic SAR
dataframe['sar'] = ta.SAR(dataframe)
# TEMA - Triple Exponential Moving Average
dataframe['tema'] = ta.TEMA(dataframe, timeperiod=9)
# Cycle Indicator
# ------------------------------------
# Hilbert Transform Indicator - SineWave
hilbert = ta.HT_SINE(dataframe)
dataframe['htsine'] = hilbert['sine']
dataframe['htleadsine'] = hilbert['leadsine']
pta.cti(dataframe.close, 100)
# Pattern Recognition - Bullish candlestick patterns
# ------------------------------------
# # Hammer: values [0, 100]
# dataframe['CDLHAMMER'] = ta.CDLHAMMER(dataframe)
# # Inverted Hammer: values [0, 100]
# dataframe['CDLINVERTEDHAMMER'] = ta.CDLINVERTEDHAMMER(dataframe)
# # Dragonfly Doji: values [0, 100]
# dataframe['CDLDRAGONFLYDOJI'] = ta.CDLDRAGONFLYDOJI(dataframe)
# # Piercing Line: values [0, 100]
# dataframe['CDLPIERCING'] = ta.CDLPIERCING(dataframe) # values [0, 100]
# # Morningstar: values [0, 100]
# dataframe['CDLMORNINGSTAR'] = ta.CDLMORNINGSTAR(dataframe) # values [0, 100]
# # Three White Soldiers: values [0, 100]
# dataframe['CDL3WHITESOLDIERS'] = ta.CDL3WHITESOLDIERS(dataframe) # values [0, 100]
# Pattern Recognition - Bearish candlestick patterns
# ------------------------------------
# # Hanging Man: values [0, 100]
# dataframe['CDLHANGINGMAN'] = ta.CDLHANGINGMAN(dataframe)
# # Shooting Star: values [0, 100]
# dataframe['CDLSHOOTINGSTAR'] = ta.CDLSHOOTINGSTAR(dataframe)
# # Gravestone Doji: values [0, 100]
# dataframe['CDLGRAVESTONEDOJI'] = ta.CDLGRAVESTONEDOJI(dataframe)
# # Dark Cloud Cover: values [0, 100]
# dataframe['CDLDARKCLOUDCOVER'] = ta.CDLDARKCLOUDCOVER(dataframe)
# # Evening Doji Star: values [0, 100]
# dataframe['CDLEVENINGDOJISTAR'] = ta.CDLEVENINGDOJISTAR(dataframe)
# # Evening Star: values [0, 100]
# dataframe['CDLEVENINGSTAR'] = ta.CDLEVENINGSTAR(dataframe)
# Pattern Recognition - Bullish/Bearish candlestick patterns
# ------------------------------------
# # Three Line Strike: values [0, -100, 100]
# dataframe['CDL3LINESTRIKE'] = ta.CDL3LINESTRIKE(dataframe)
# # Spinning Top: values [0, -100, 100]
# dataframe['CDLSPINNINGTOP'] = ta.CDLSPINNINGTOP(dataframe) # values [0, -100, 100]
# # Engulfing: values [0, -100, 100]
# dataframe['CDLENGULFING'] = ta.CDLENGULFING(dataframe) # values [0, -100, 100]
# # Harami: values [0, -100, 100]
# dataframe['CDLHARAMI'] = ta.CDLHARAMI(dataframe) # values [0, -100, 100]
# # Three Outside Up/Down: values [0, -100, 100]
# dataframe['CDL3OUTSIDE'] = ta.CDL3OUTSIDE(dataframe) # values [0, -100, 100]
# # Three Inside Up/Down: values [0, -100, 100]
# dataframe['CDL3INSIDE'] = ta.CDL3INSIDE(dataframe) # values [0, -100, 100]
# # Chart type
# # ------------------------------------
# # Heikin Ashi Strategy
# heikinashi = qtpylib.heikinashi(dataframe)
# dataframe['ha_open'] = heikinashi['open']
# dataframe['ha_close'] = heikinashi['close']
# dataframe['ha_high'] = heikinashi['high']
# dataframe['ha_low'] = heikinashi['low']
# Retrieve best bid and best ask from the orderbook
# ------------------------------------
"""
# first check if dataprovider is available
if self.dp:
if self.dp.runmode.value in ('live', 'dry_run'):
ob = self.dp.orderbook(metadata['pair'], 1)
dataframe['best_bid'] = ob['bids'][0][0]
dataframe['best_ask'] = ob['asks'][0][0]
"""
return dataframe
def populate_entry_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
"""
Based on TA indicators, populates the entry signal for the given dataframe
:param dataframe: DataFrame
:param metadata: Additional information, like the currently traded pair
:return: DataFrame with entry columns populated
"""
dataframe.loc[
(
# Signal: RSI crosses above 30
# (qtpylib.crossed_above(dataframe['rsi'], self.buy_rsi.value)) &
# (dataframe['tema'] <= dataframe['bb_middleband']) & # Guard: tema below BB middle
(dataframe['tema'] > dataframe['tema'].shift(1)) & # Guard: tema is raising
(dataframe['volume'] > 0) # Make sure Volume is not 0
),
'enter_long'] = 1
dataframe.loc[
(
# Signal: RSI crosses above 70
# (qtpylib.crossed_above(dataframe['rsi'], self.short_rsi.value)) &
# (dataframe['tema'] > dataframe['bb_middleband']) & # Guard: tema above BB middle
(dataframe['tema'] < dataframe['tema'].shift(1)) & # Guard: tema is falling
(dataframe['volume'] > 0) # Make sure Volume is not 0
),
'enter_short'] = 1
return dataframe
def populate_exit_trend(self, dataframe: DataFrame, metadata: dict) -> DataFrame:
"""
Based on TA indicators, populates the exit signal for the given dataframe
:param dataframe: DataFrame
:param metadata: Additional information, like the currently traded pair
:return: DataFrame with exit columns populated
"""
dataframe.loc[
(
# Signal: RSI crosses above 70
(qtpylib.crossed_above(dataframe['rsi'], self.sell_rsi.value)) &
# (dataframe['tema'] > dataframe['bb_middleband']) & # Guard: tema above BB middle
(dataframe['tema'] < dataframe['tema'].shift(1)) & # Guard: tema is falling
(dataframe['volume'] > 0) # Make sure Volume is not 0
),
'exit_long'] = 1
dataframe.loc[
(
# Signal: RSI crosses above 30
(qtpylib.crossed_above(dataframe['rsi'], self.exit_short_rsi.value)) &
# Guard: tema below BB middle
# (dataframe['tema'] <= dataframe['bb_middleband']) &
(dataframe['tema'] > dataframe['tema'].shift(1)) & # Guard: tema is raising
(dataframe['volume'] > 0) # Make sure Volume is not 0
),
'exit_short'] = 1
return dataframewell indicators disables some warnings (warnings.simplefilter(action="ignore", category=RuntimeWarning)) ...
so the most likely reason is that the warnings are always there - just hyperopt is not disabling these warnings in some areas.
italodamato
commented
2 years ago yep, that's what I was worried about...
xmatthias
commented
2 years ago I think your best bet for this will be to report this to pandas_ta - maybe they have some suggestions or reasons as to why this happens.
You'll probably have to strip it from freqtrade code - but we do provide methods to load the data directly (maybe into a notebook) - so this can be tested and shown detached from freqtrade code.
xmatthias
commented
2 years ago Any news on this? knowing you a bit i'm sure you've investigated this further - so i'm curious as to what the outcome was.
i do think that the warning is always there (independent of hyperopt / the hyperopt mode) - but qtpylib is hiding it in most instances (not in hyperopt - but that's easily possible due to the way hyperopt works).
italodamato
commented
2 years ago I haven't asked them but, from a quick check, it happens on LUNA-USDT-4h when the length is set to 1 and it happens only the first time I run it (I guess the warning gets disabled if I try again).
The point is that I wasn't using length=1 as a parameter in my strategy so it's either something that gets used under the hood (maybe for some checks?) or there's something else happening when using analyze-per-epoch
xmatthias
commented
2 years ago the error comes from linreg - which is used by several other indicators.
wasn't luna suspended for a time? maybe it's the missing candles (near or at 0) that cause problems?
xmatthias
commented
1 year ago Unfortunately, this is not something we will be able to fix here.
Errors are surpressed when importing qtpylib.indicators - but the error itself is coming from pandas_ta - so this is best investigated with them.
Describe your environment
Describe the problem:
getting
RuntimeWarning: invalid value encountered in double_scalarsinpandas_ta.ctiwhen hyperopting using--analyze-per-epochhttps://github.com/twopirllc/pandas-ta/blob/96555b8752061405733664bb2f61d09cac9b29fd/pandas_ta/momentum/cti.py
Steps to reproduce:
pandas_ta.cti--analyze-per-epochObserved Results:
RuntimeWarning: invalid value encountered in double_scalars--analyze-per-epochRelevant code exceptions or logs