search

nateemma / strategies

Custom trading strategies using the freqtrade framework
329 stars 88 forks source link

Several errors / warnings on the strategy groups #24

Open Jarrodsz opened 1 year ago

Jarrodsz commented 1 year ago

Im running several of the strats in a devcontainer to not much avail. For a group of the first 10 NNTC* I let it run overnight with the following warnings all the time and not a single buy/sell.


    predicting buys/sells...
Testing Classifier ( GRU )
/home/ftuser/.local/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/home/ftuser/.local/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
/home/ftuser/.local/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, msg_start, len(result))
              precision    recall  f1-score   support

           0       0.83      1.00      0.90       151
           1       0.00      0.00      0.00        19
           2       0.00      0.00      0.00        13

    accuracy                           0.83       183
   macro avg       0.28      0.33      0.30       183
weighted avg       0.68      0.83      0.75       183

Or

.//binance/NNPredict_TFT.log:2023-05-03 02:32:15,159 - freqtrade.strategy.strategy_wrapper - ERROR - Unexpected error Trainer.__init__() got an unexpected keyword argument 'auto_lr_find' calling <bound method IStrategy._analyze_ticker_internal of <NNPredict_TFT.NNPredict_TFT object at 0xffff71cabd90>>

On the NNTC I get error that the model files cannot be found. While the file on disk exists.

Perhaps im missing some dependencies?

just-nilux commented 1 year ago

You seem to be new! Try this 🥇

As for TFT... How about you start to learn about some less complex models first, it helps later on in understanding...

tomjrtsmith commented 1 year ago

Try some of the older PCA strats and have a read through the code see what's happening, there's a bunch of libraries you'll need

nateemma commented 1 year ago

The UndefinedMetricWarning warning means that zero buy/sell signals were generated, so the F1 metric is undefined. It's not exactly an error, it just means the algorithm is not producing anything (some of them do this). Which strat was it (NNTC_macd_GRU?)?

As an aside, these are the results I got recently (over 30 days):

+-------------------------+----------+------------+----------+--------+--------------+--------+

| Strategy | Trades | Average% | Total% | Win% | Expectancy | Rank |

|-------------------------+----------+------------+----------+--------+--------------+--------|

| NNTC_profit_Wavenet | 64 | 0.34 | 6.49 | 81.2 | 0.1 | 1 |

| NNTC_macd_Multihead | 78 | 0.2 | 4.62 | 82.1 | 0.06 | 2 |

| NNTC_profit_LSTM2 | 31 | 0.41 | 3.85 | 80.6 | 0.28 | 3 |

| NNTC_profit_Transformer | 73 | 0.06 | 1.4 | 79.5 | 0.02 | 4 |

| NNTC_dwt_LSTM | 27 | 0.03 | 0.23 | 88.9 | 0.01 | 5 |

| NNTC_nseq_Ensemble | 0 | 0 | 0 | 0 | 0 | 6 |

| NNTC_macd_GRU | 0 | 0 | 0 | 0 | 0 | 6 |

| NNTC_nseq_Transformer | 0 | 0 | 0 | 0 | 0 | 6 |

| NNTC_nseq_Wavenet | 0 | 0 | 0 | 0 | 0 | 6 |

| NNTC_macd_Transformer | 0 | 0 | 0 | 0 | 0 | 6 |

| NNTC_nseq_GRU | 0 | 0 | 0 | 0 | 0 | 6 |

| NNTC_dwt_Wavenet | 0 | 0 | 0 | 0 | 0 | 6 |

| NNTC_pv_Wavenet | 0 | 0 | 0 | 0 | 0 | 6 |

| NNTC_dwt_CNN | 22 | -0.03 | -0.17 | 77.3 | -0.02 | 14 |

| NNTC_profit_CNN | 35 | -0.05 | -0.51 | 74.3 | -0.02 | 15 |

| NNTC_fbb_GRU | 59 | -0.1 | -1.72 | 81.4 | -0.02 | 16 |

| NNTC_nseq_LSTM | 50 | -0.2 | -3.02 | 68 | -0.06 | 17 |

| NNTC_swing_LSTM | 1 | -10.82 | -3.25 | 0 | 0 | 18 |

| NNTC_profit_Wavenet2 | 57 | -0.21 | -3.56 | 71.9 | -0.07 | 19 |

| NNTC_highlow_LSTM | 86 | -0.16 | -4.01 | 77.9 | -0.04 | 20 |

| NNTC_profit_LSTM3 | 50 | -0.38 | -5.71 | 46 | -0.37 | 21 |

| NNTC_profit_GRU | 56 | -0.35 | -5.93 | 71.4 | -0.1 | 22 |

| NNTC_highlow_Ensemble | 130 | -0.18 | -6.95 | 83.8 | -0.04 | 23 |

| NNTC_profit_MLP | 256 | -0.1 | -7.33 | 37.1 | -0.16 | 24 |

| NNTC_profit_Ensemble | 121 | -0.21 | -7.46 | 76.9 | -0.06 | 25 |

| NNTC_macd_LSTM | 86 | -0.3 | -7.66 | 72.1 | -0.07 | 26 |

| NNTC_fbb_Wavenet | 40 | -0.68 | -8.18 | 70 | -0.13 | 27 |

| NNTC_pv_LSTM | 95 | -0.41 | -11.56 | 78.9 | -0.09 | 28 |

| NNTC_profit_LSTM | 42 | -0.97 | -12.26 | 59.5 | -0.28 | 29 |

| NNTC_pv_Multihead | 135 | -0.32 | -12.79 | 40.7 | -0.36 | 30 |

| NNTC_fbb_Multihead | 86 | -0.58 | -14.92 | 74.4 | -0.13 | 31 |

| NNTC_fbb_Ensemble | 97 | -0.52 | -15.27 | 75.3 | -0.11 | 32 |

| NNTC_fbb_LSTM | 85 | -0.65 | -16.56 | 72.9 | -0.13 | 33 |

+-------------------------+----------+------------+----------+--------+--------------+--------+

The second error is for NNPredict_TFT, which uses darts and pytorch libraries. The model file isn't there because it's too large for me to upload to github (6 GB). I would not recommend using the NNPredict strats, I don't think they work properly. They actually perform ok in testing, but if you look at the actual predictions it doesn't look right.

Also, take a look at user_data/strategies/scripts/install_packages.sh, I use that to update packages after updating freqtrade

Hope that helps

Cheers,

Phil

On Tue, May 2, 2023 at 11:39 PM Jarrodsz @.***> wrote:

Im running several of the strats in a devcontainer to not much avail. For a group of the first 10 NNTC* I let it run overnight with the following warnings all the time and not a single buy/sell.

predicting buys/sells...

Testing Classifier ( GRU ) /home/ftuser/.local/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use zero_division parameter to control this behavior. _warn_prf(average, modifier, msg_start, len(result)) /home/ftuser/.local/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use zero_division parameter to control this behavior. _warn_prf(average, modifier, msg_start, len(result)) /home/ftuser/.local/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1334: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use zero_division parameter to control this behavior. _warn_prf(average, modifier, msg_start, len(result)) precision recall f1-score support

       0       0.83      1.00      0.90       151
       1       0.00      0.00      0.00        19
       2       0.00      0.00      0.00        13

accuracy                           0.83       183

macro avg 0.28 0.33 0.30 183 weighted avg 0.68 0.83 0.75 183

Or

.//binance/NNPredict_TFT.log:2023-05-03 02:32:15,159 - freqtrade.strategy.strategy_wrapper - ERROR - Unexpected error Trainer.init() got an unexpected keyword argument 'auto_lr_find' calling <bound method IStrategy._analyze_ticker_internal of <NNPredict_TFT.NNPredict_TFT object at 0xffff71cabd90>>

On the NNTC I get error that the model files cannot be found. While the file on disk exists.

Perhaps im missing some dependencies?

— Reply to this email directly, view it on GitHub https://github.com/nateemma/strategies/issues/24, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X57C6EEXODVJCS3Y7Y3XEH4RTANCNFSM6AAAAAAXT7CCUM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

just-nilux commented 1 year ago

Interesting results. I'm using NNTC with LSTM3 and autoencoder instead of scitkit pca and have quite good results! I feel that using autoencoder delivers better results if you have more features. But this is just a first impression....

Have you tried using Anomaly with NBeats? I've read so many good things about NBeats but didn't have the time looking into it more.

Overall a note regarding performance: It improved significantly since I introduced shifted candles to my feature set. Also, try the SMI (stochastic momentum) instead of MFI to validate signals.

nateemma commented 1 year ago

I haven't tried it with NNTC, but I already have a classifier written for NNPredict, so should be pretty easy. I'll try both of these out when I have some time (which could be a while!)

Cheers,

Phil

On Thu, May 4, 2023 at 2:54 AM nilux @.***> wrote:

Interesting results. I'm using NNTC with LSTM3 and autoencoder instead of scitkit pca and have quite good results! Have you tried using NNTC with NBeats? I've read so many good things about NBeats but didn't have the time looking into it more.

— Reply to this email directly, view it on GitHub https://github.com/nateemma/strategies/issues/24#issuecomment-1534443694, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X55MZIYL2ACA55V2LXDXEN4D3ANCNFSM6AAAAAAXT7CCUM . You are receiving this because you commented.Message ID: @.***>