Poor model performance

[Oussama954]

I attempted to train NNTC models using a 1-2 year history for a specific pair. However, the model stops after 10-15 minutes with early stopping, and the output is not satisfactory. For example, during epoch 80/256, the loss, accuracy, and mse are 0.1515, 0.9404, and 0.0285 respectively, while the validation loss, validation accuracy, and validation mse are 2.8903, 0.6009, and 0.2273. I have tried all NNTC.py models, but without success. What could I be doing wrong?

[nateemma]

The early stopping isn't really a problem - for a single pair, most of them should converge in less than 100 iterations. However, your loss values do not look good at all. Have you tried running hyperopt? It can make a huge difference in performance.

The other thing that can make a difference is the pairlist - if you include unstable pairs, or pairs that do not have a long history, the algorithms will likely not do well.

Can you send me the full output for one of the strategies please?

Thanks,

Phil

On Sun, May 7, 2023 at 6:18 AM Oussama954 @.***> wrote:

I attempted to train NNTC models using a 1-2 year history for a specific pair. However, the model stops after 10-15 minutes with early stopping, and the output is not satisfactory. For example, during epoch 80/256, the loss, accuracy, and mse are 0.1515, 0.9404, and 0.0285 respectively, while the validation loss, validation accuracy, and validation mse are 2.8903, 0.6009, and 0.2273. I have tried all NNTC.py models, but without success. What could I be doing wrong? [image: image] https://user-images.githubusercontent.com/7197617/236679817-529f4a67-a3c6-448e-8243-30b44c0329c0.png

— Reply to this email directly, view it on GitHub https://github.com/nateemma/strategies/issues/26, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X52PC6FWMDB2LUXWMV3XE6OJTANCNFSM6AAAAAAXY37TH4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Oussama954]

Thank you for your response, Phil. Yes, I have tried running hyperopt, but unfortunately, it did not improve the model's performance in this case. Regarding the pairlist, I have made sure to include pairs with a long history for this specific training, but without success

I have attached the full log for one of the strategies as requested. Please let me know if you have any other suggestions or if there is any other information that may be helpful in identifying the issue.

Thank you output.txt

[nateemma]

OK, the number of buy/sell signals looks very wrong - there are more buy or sell signals than holds:

          precision    recall  f1-score   support

       0       0.31      0.34      0.32      1913
       1       0.49      0.55      0.52      3041
       2       0.59      0.50      0.54      3699

Here, 0 is hold, 1 is buy, 2 is sell.

There are so many buy/sell events that the neural network will not likely be able to converge on a pattern. This is also indicated by the fact that you did not get many trades (9 in 6 months).

Did you change the thresholds in the strategy file (NNTC_profit_Wavenet.py)?

Thanks,

Phil

On Sun, May 7, 2023 at 8:21 AM Oussama954 @.***> wrote:

Thank you for your response, Phil. Yes, I have tried running hyperopt, but unfortunately, it did not improve the model's performance in this case. Regarding the pairlist, I have made sure to include pairs with a long history for this specific training, but without success

I have attached the full log for one of the strategies as requested. Please let me know if you have any other suggestions or if there is any other information that may be helpful in identifying the issue.

Thank you output.txt https://github.com/nateemma/strategies/files/11415339/output.txt

— Reply to this email directly, view it on GitHub https://github.com/nateemma/strategies/issues/26#issuecomment-1537466936, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X57QOR5RPNX4MINFWZ3XE64W3ANCNFSM6AAAAAAXY37TH4 . You are receiving this because you commented.Message ID: @.***>

[Oussama954]

To answer your question, I have not changed the thresholds in the strategy file (NNTC_profit_Wavenet.py). The only modification I made was to the list of indicator in DataframePopulator.py. As a side , I would like to mention that i am using Binance data for training

[Jarrodsz]

The early stopping isn't really a problem - for a single pair, most of them should converge in less than 100 iterations. However, your loss values do not look good at all. Have you tried running hyperopt? It can make a huge difference in performance. The other thing that can make a difference is the pairlist - if you include unstable pairs, or pairs that do not have a long history, the algorithms will likely not do well. Can you send me the full output for one of the strategies please? Thanks, Phil …

Just to chime in. What paiist filters do You advice or what paiist should work good in general? Im using binance not binanceus and also had similar bad results so far. Probably tackling this from the bottom with a decent pairlist would be the start of improving?

[nateemma]

Ah, I see you have 180 indicators. Those are 'compressed' to 64, which is likely too much compression.

Can you run PCA_profit on the same data and send me the output please? That will give me an idea of what the compression should be for your collection of indicators

Thanks,

Phil

On Sun, May 7, 2023 at 8:53 AM Oussama954 @.***> wrote:

To answer your question, I have not changed the thresholds in the strategy file (NNTC_profit_Wavenet.py). The only modification I made was to the list of indicator in DataframePopulator.py. As a side , I would like to mention that i am using Binance data for training

— Reply to this email directly, view it on GitHub https://github.com/nateemma/strategies/issues/26#issuecomment-1537475764, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X56OFUWRNKJRHYJLG5TXE7APNANCNFSM6AAAAAAXY37TH4 . You are receiving this because you commented.Message ID: @.***>

[Oussama954]

Thank you for your suggestion, Phil. However, I don't think the issue is related to the compression of the indicators. I have tested the model without any changes to the indicator list and the results were consistently poor.

Given that the data is sourced from the Binance , it is possible that the data may be contributing to the poor performance of the model ?

I have attached the full log without any modification in the code! output.txt

[nateemma]

Those results are significantly better, so the compression is certainly having some effect. I'm still confused as to why there are so many buy/sell events compared to the binanceus version though.

In NNTC_profit_Wavenet.py, try modifying these lines:

n_profit_stddevs = 2.0 n_loss_stddevs = 2.0

Increase these to 3.0 and see what happens

Thanks,

Phil

P.S. the "profit" variants do not perform particularly well on binanceus either, the "macd" variants seem to be consistently better

On Sun, May 7, 2023 at 9:34 AM Oussama954 @.***> wrote:

Thank you for your suggestion, Phil. However, I don't think the issue is related to the compression of the indicators. I have tested the model without any changes to the indicator list and the results were consistently poor.

Given that the data is sourced from the Binance , it is possible that the data may be contributing to the poor performance of the model ?

I have attached the full log without any modification in the code! output.txt https://github.com/nateemma/strategies/files/11415516/output.txt

— Reply to this email directly, view it on GitHub https://github.com/nateemma/strategies/issues/26#issuecomment-1537486992, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X54VAG2TX4UV3MIL6DLXE7FKTANCNFSM6AAAAAAXY37TH4 . You are receiving this because you commented.Message ID: @.***>

[nateemma]

you can see my list in user_data/strategies/binanceus/config_binanceus.json

I use the following command to generate that list:

freqtrade test-pairlist --exchange binanceus -c user_data/strategies/binanceus/config_binanceus_top.json

if you just change binanceus -> binance, it should work for you too (but remember to change single quotes to double quotes if you paste the list into your config file)

Thanks,

Phil

On Sun, May 7, 2023 at 9:10 AM Jarrodsz @.***> wrote:

The early stopping isn't really a problem - for a single pair, most of them should converge in less than 100 iterations. However, your loss values do not look good at all. Have you tried running hyperopt? It can make a huge difference in performance. The other thing that can make a difference is the pairlist - if you include unstable pairs, or pairs that do not have a long history, the algorithms will likely not do well. Can you send me the full output for one of the strategies please? Thanks, Phil … <#m-3576039310694674804>

Just to chime in. What paiist filters do You advice or what paiist should work good in general? Im using binance not binanceus and also had similar bad results so far. Probably tackling this from the bottom with a decent pairlist would be the start of improving?

— Reply to this email directly, view it on GitHub https://github.com/nateemma/strategies/issues/26#issuecomment-1537480680, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X55UVJTDWOFNFXZKDKDXE7CP3ANCNFSM6AAAAAAXY37TH4 . You are receiving this because you commented.Message ID: @.***>

[just-nilux]

@Oussama954 This is related to the feature set and compression. You can try to set compression to false to disable it completely.

In general it helps if you do the feature engineering step by step. Start with very little and increase. More indicator are usually bad for the performs. A few good ones that are related to your target is the way to go.

[nateemma]

FYI, I found an issue with the calculation of profit thresholds. Working on it...

Thanks,

Phil

On Sun, May 7, 2023 at 2:19 PM nilux @.***> wrote:

@Oussama954 https://github.com/Oussama954 This is related to the feature set and compression. You can try to set compression to false to disable it completely.

In general it helps if you do the feature engineering step by step. Start with very little and increase. More indicator are usually bad for the performs. A few good ones that are related to your target is the way to go.

— Reply to this email directly, view it on GitHub https://github.com/nateemma/strategies/issues/26#issuecomment-1537542936, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X57J5OOTSKEA3KCRU4TXFAGUNANCNFSM6AAAAAAXY37TH4 . You are receiving this because you commented.Message ID: @.***>

[just-nilux]

FYI, I found an issue with the calculation of profit thresholds. Working on it... Thanks, Phil … On Sun, May 7, 2023 at 2:19 PM nilux @.> wrote: @Oussama954 https://github.com/Oussama954 This is related to the feature set and compression. You can try to set compression to false to disable it completely. In general it helps if you do the feature engineering step by step. Start with very little and increase. More indicator are usually bad for the performs. A few good ones that are related to your target is the way to go. — Reply to this email directly, view it on GitHub <#26 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X57J5OOTSKEA3KCRU4TXFAGUNANCNFSM6AAAAAAXY37TH4 . You are receiving this because you commented.Message ID: @.>

Just an advise: It would be better if you use one specific issue to post these things so we don't need to look at all issues to see your updates :)