nateemma
commented
1 year ago Hi,
a couple of comments:
- you are correct in that some of these files are not strategies. You should only run files of the form PCA*.py, Anomaly.py, NNTC_.py
- if you look in user_data/strategies/scripts/overnight.sh, you will find some shell code that builds the list for you. You supply the pattern (PCA, Anomaly, NNTC etc) and it builds an array of the files present and does something with them (in this case run hyperopt and backtest). This might be easier (unless you want to 'curate' the list)
- as an aside, there is a python script called SummariseTestResults.py that will parse the output log from overnight.sh (or any backtest output) and provide a table of summarised results
- I don't really test this way. I tend to run overnight.sh on each group and select the ones that do best, and then run dry-run on those. I like to run each dry-run in a separate terminal, and at the same time, and I run them for several days/weeks. I do it this way because I like to see the output from each strat separately, plus I can easily stop/restart any of the strats. Also, be careful starting several strats at once as the exchange is likely to complain.
Hope that helps,
Cheers,
Phil
On Sat, Apr 29, 2023 at 3:53 PM Jarrodsz @.***> wrote:
This repro is one of very much inspiration and much gratitude to share this. Im relative new to freqtrade, strats and ml so im learning each day.
Since its quite a task to start all the scripts manually and the startup scripts did not seem to provide a way of starting them all I started a little WIP startup script to start groups of strats together for testing. It seems to work as long as you do not provide an api key in config. Below is a wip and you would need the rest of the config but it gives an idea, is this a good idea of interest to anyone?
!/bin/bash
cd /freqtrade
define the strats from port 8100 and up
declare -a ANOMALY_STRATS=( "binance usdt 8100 Anomaly" "binance usdt 8101 AnomalyDetector" "binance usdt 8102 AnomalyDetector_AEnc" "binance usdt 8103 AnomalyDetector_DBSCAN" "binance usdt 8104 AnomalyDetector_EE" "binance usdt 8105 AnomalyDetector_Ensemble" "binance usdt 8106 AnomalyDetector_GMix" "binance usdt 8107 AnomalyDetector_IFOR" "binance usdt 8108 AnomalyDetector_KMeans" "binance usdt 8109 AnomalyDetector_LOF" "binance usdt 8110 AnomalyDetector_LSTM" "binance usdt 8111 AnomalyDetector_PCA" "binance usdt 8112 AnomalyDetector_SVM" "binance usdt 8113 Anomaly_dwt" "binance usdt 8114 Anomaly_fbb" "binance usdt 8115 Anomaly_highlow" "binance usdt 8116 Anomaly_macd" "binance usdt 8117 Anomaly_nseq" "binance usdt 8118 Anomaly_highlow" "binance usdt 8119 Anomaly_profit" "binance usdt 8120 Anomaly_pv" )
declare -a PWA_STRATS=( "binance usdt 8100 AnomalyDetector_PCA" "binance usdt 8101 PCA" "binance usdt 8102 PCA_dwt" "binance usdt 8103 PCA_fbb" "binance usdt 8104 PCA_fwr" "binance usdt 8105 PCA_highlow" "binance usdt 8106 PCA_jump" "binance usdt 8107 PCA_macd" "binance usdt 8108 PCA_mfi" "binance usdt 8109 PCA_minmax" "binance usdt 8110 PCA_nseq" "binance usdt 8111 PCA_over" "binance usdt 8112 PCA_profit" "binance usdt 8113 PCA_pv" "binance usdt 8114 PCA_stochastic" "binance usdt 8115 PCA_swing" )
declare -a NNTC_STRATS=( "binance usdt 8100 NNTC" "binance usdt 8101 NNTC_dwt_CNN" "binance usdt 8102 NNTC_dwt_LSTM" "binance usdt 8103 NNTC_dwt_Wavenet" "binance usdt 8104 NNTC_fbb_Ensemble" "binance usdt 8105 NNTC_fbb_GRU" "binance usdt 8106 NNTC_fbb_LSTM" "binance usdt 8107 NNTC_fbb_Multihead" "binance usdt 8108 NNTC_fbb_Wavenet" "binance usdt 8109 NNTC_fwr_LSTM" "binance usdt 8110 NNTC_highlow_Ensemble" "binance usdt 8111 NNTC_highlow_LSTM" "binance usdt 8112 NNTC_macd_GRU" "binance usdt 8113 NNTC_macd_LSTM" "binance usdt 8114 NNTC_macd_Multihead" "binance usdt 8115 NNTC_macd_Transformer" "binance usdt 8116 NNTC_nseq_Ensemble" "binance usdt 8117 NNTC_nseq_GRU" "binance usdt 8118 NNTC_nseq_LSTM" "binance usdt 8119 NNTC_nseq_Transformer" "binance usdt 8120 NNTC_nseq_Wavenet" "binance usdt 8121 NNTC_profit_CNN" "binance usdt 8122 NNTC_profit_Ensemble" "binance usdt 8123 NNTC_profit_GRU" "binance usdt 8124 NNTC_profit_LSTM" "binance usdt 8125 NNTC_profit_LSTM2" "binance usdt 8126 NNTC_profit_LSTM3" "binance usdt 8127 NNTC_profit_MLP" "binance usdt 8128 NNTC_profit_Transformer" "binance usdt 8129 NNTC_profit_Wavenet" "binance usdt 8130 NNTC_profit_Wavenet2" "binance usdt 8131 NNTC_pv_LSTM" "binance usdt 8132 NNTC_pv_Multihead" "binance usdt 8133 NNTC_pv_Wavenet" "binance usdt 8134 NNTC_swing_LSTM" "binance usdt 8135 NNTClassifier_CNN" "binance usdt 8136 NNTClassifier_Ensemble" "binance usdt 8137 NNTClassifier_GRU" "binance usdt 8138 NNTClassifier_LSTM" "binance usdt 8139 NNTClassifier_LSTM2" "binance usdt 8140 NNTClassifier_LSTM3" )
Set the exchange variable
exchange="binance"
market is not used for now
but could be used to load a specifc market pair like ETH / BTC
Merge the two arrays
declare -a STRATS=("${PWA_STRATS[@]}")
declare -a STRATS=("${PWA_STRATS[@]}" "${NNTC_STRATS[@]}")
Loop through the STRATS array and run the freqtrade trade command for each strategy
for cmd in "${STRATS[@]}" do exchange="$(echo "${cmd}" | cut -d' ' -f1)" market="$(echo "${cmd}" | cut -d' ' -f2)" port="$(echo "${cmd}" | cut -d' ' -f3)" strategy="$(echo "${cmd}" | cut -d' ' -f4)"
cat << END
$(date +%F\ %T) Dry-run strategy:${strategy} for exchange:${exchange} on market:${market} with port:${port}...
END
export FREQTRADE__API_SERVER__LISTEN_PORT="${port}" freqtrade trade --dry-run \ --config /freqtrade/user_data/config/default.json \ --config /freqtrade/user_data/config/exchange/"${exchange}".json \ --db-url sqlite:////freqtrade/user_data/database/"${exchange}"/"${strategy}"_strategy.sqlite \ -s "${strategy}" &
sleep 30
done
This could ofcourse be a bit more extended perhaps with a question prompt on what group of strats to test. I modified the dry_run.sh scripts a bit and added some more easy way to define and start them. If anyone interested I will share a full repro of this. I did blend it with my own config of freqtrade splitting up some of the config files. Currently I have all strats in user_data/strategies so it could be refined a it more but this is just an iniitial idea.
The --config /freqtrade/user_data/config/exchange/"${exchange}".json \ ( binance.json ) for example loads some specific config files to easy the number or arguments in the shell to start the bot:
{ "exchange": { "name": "binance", "key": "", "secret": "", "ccxt_config": { "enableRateLimit": true, "rateLimit": 195, "urls": { "api": { "public": "http://binance_proxy_nightshift2k:8090/api/v3" } } }, "ccxt_async_config": { "enableRateLimit": true, "rateLimit": 195 } }, "add_config_files": [ "/freqtrade/user_data/config/pairlist/static.json", "/freqtrade/user_data/config/includes/pricing.json", "/freqtrade/user_data/config/exchange/binance/pairlist.json", "/freqtrade/user_data/config/exchange/binance/blacklist.json", "/freqtrade/user_data/config/includes/frequi.json" ] }
Hope to learn a lot and studying the strategies has given me real new insights on how to approach ML. It would be awesome if freq ai could be used in the near future
QUESTION: I just included in the array most of the strats probably some helper files base class files in there that need to be removed. My goal is to put only the most promising strats in the array and drop the ones that are not performing then try to study the good ones learn and help in improving them.
Kind regards and thank you for providing this awesome insight into ML with freqtrade <3
— Reply to this email directly, view it on GitHub https://github.com/nateemma/strategies/issues/21, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABD4X555Z2IQBTNZ2AUVBQLXDWLVLANCNFSM6AAAAAAXQP4RQ4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>
This repro is one of very much inspiration and much gratitude to share this. Im relative new to freqtrade, strats and ml so im learning each day.
Since its quite a task to start all the scripts manually and the startup scripts did not seem to provide a way of starting them all I started a little WIP startup script to start groups of strats together for testing. It seems to work as long as you do not provide an api key in config. Below is a wip and you would need the rest of the config but it gives an idea, is this a good idea of interest to anyone?
This could ofcourse be a bit more extended perhaps with a question prompt on what group of strats to test. I modified the dry_run.sh scripts a bit and added some more easy way to define and start them. If anyone interested I will share a full repro of this. I did blend it with my own config of freqtrade splitting up some of the config files. Currently I have all strats in user_data/strategies so it could be refined a it more but this is just an iniitial idea.
The
--config /freqtrade/user_data/config/exchange/"${exchange}".json \( binance.json ) for example loads some specific config files to easy the number or arguments in the shell to start the bot:Hope to learn a lot and studying the strategies has given me real new insights on how to approach ML. It would be awesome if freq ai could be used in the near future
QUESTION: I just included in the array most of the strats probably some helper files base class files in there that need to be removed. My goal is to put only the most promising strats in the array and drop the ones that are not performing then try to study the good ones learn and help in improving them.
I run this trade.sh from the user_data folder due to mounting issues ( more easy ) they are then run in 1 single docker container, using multiple docker containers with a docker-compose resulted in a lot more cpu and memory usage.
Kind regards and thank you for providing this awesome insight into ML with freqtrade <3