freqtrade / freqtrade

Free, open source crypto trading bot
https://www.freqtrade.io
GNU General Public License v3.0
28.31k stars 6.07k forks source link

Process Died ... ddosp problems Binance impact on "production" bot #1805

Closed Prossi79 closed 5 years ago

Prossi79 commented 5 years ago

Step 1: Have you search for this issue before posting it?

Step 2: Describe your environment

Step 3: Describe the problem:

I have tried to fetch data for backtesting on Binance ... 1m, probably too much

My bot on Binance running in production mode "died" short after, the one in dry run not

Steps to reproduce:

Observed Results:

I think the dead process was caused by the DDoSP of Binance ... but the bot should not stop, if I overload Binance by my requests, right?

Relevant code exceptions or logs:

 2019-04-25 12:07:27,614 - freqtrade.pairlist.IPairList - INFO - Ignoring BSV/BTC from whitelist. Market is not active.
2019-04-25 12:07:27,679 - freqtrade.exchange.exchange - WARNING - _async_get_candle_history() returned exception: "Could not load ticker history due to RequestTimeout. Message: binance GET https://api.binance.com/api/v1/klines?symbol=XLMBTC&interval=5m "
2019-04-25 12:07:27,680 - freqtrade.exchange.exchange - WARNING - Giving up retrying: _async_get_candle_history()
2019-04-25 12:07:27,680 - freqtrade.worker - WARNING - Error: Could not load ticker history due to RequestTimeout. Message: binance GET https://api.binance.com/api/v1/klines?symbol=XLMBTC&interval=5m , retrying in 30 seconds...
2019-04-25 12:07:57,709 - freqtrade.pairlist.IPairList - INFO - Ignoring BSV/BTC from whitelist. Market is not active.
2019-04-25 12:07:57,748 - freqtrade.exchange.exchange - WARNING - _async_get_candle_history() returned exception: "Could not load ticker history due to RequestTimeout. Message: binance GET https://api.binance.com/api/v1/klines?symbol=QTUMBTC&interval=5m "
2019-04-25 12:07:57,749 - freqtrade.exchange.exchange - WARNING - Giving up retrying: _async_get_candle_history()
2019-04-25 12:07:57,749 - freqtrade.worker - WARNING - Error: Could not load ticker history due to RequestTimeout. Message: binance GET https://api.binance.com/api/v1/klines?symbol=QTUMBTC&interval=5m , retrying in 30 seconds...
2019-04-25 12:08:27,777 - freqtrade.pairlist.IPairList - INFO - Ignoring BSV/BTC from whitelist. Market is not active.
2019-04-25 12:08:27,783 - freqtrade.exchange.exchange - WARNING - _async_get_candle_history() returned exception: "Could not load ticker history due to RequestTimeout. Message: binance GET https://api.binance.com/api/v1/klines?symbol=DASHBTC&interval=5m "
2019-04-25 12:08:27,783 - freqtrade.exchange.exchange - WARNING - Giving up retrying: _async_get_candle_history()
2019-04-25 12:08:27,784 - freqtrade.worker - WARNING - Error: Could not load ticker history due to RequestTimeout. Message: binance GET https://api.binance.com/api/v1/klines?symbol=DASHBTC&interval=5m , retrying in 30 seconds...
2019-04-25 12:08:57,829 - freqtrade.pairlist.IPairList - INFO - Ignoring BSV/BTC from whitelist. Market is not active.
2019-04-25 12:08:57,835 - freqtrade.exchange.exchange - WARNING - _async_get_candle_history() returned exception: "Could not load ticker history due to RequestTimeout. Message: binance GET https://api.binance.com/api/v1/klines?symbol=XEMBTC&interval=5m "
2019-04-25 12:08:57,835 - freqtrade.exchange.exchange - WARNING - Giving up retrying: _async_get_candle_history()
2019-04-25 12:08:57,836 - freqtrade.worker - WARNING - Error: Could not load ticker history due to RequestTimeout. Message: binance GET https://api.binance.com/api/v1/klines?symbol=XEMBTC&interval=5m , retrying in 30 seconds...
2019-04-25 12:09:27,865 - freqtrade.pairlist.IPairList - INFO - Ignoring BSV/BTC from whitelist. Market is not active.
2019-04-25 12:09:30,228 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "./freqtrade/main.py", line 42, in main
    worker.run()
  File "/root/freqtrade/freqtrade/worker.py", line 72, in run
    state = self._worker(old_state=state)
  File "/root/freqtrade/freqtrade/worker.py", line 110, in _worker
    self._throttle(func=self._process, min_secs=throttle_secs)
  File "/root/freqtrade/freqtrade/worker.py", line 123, in _throttle
    result = func(*args, **kwargs)
  File "/root/freqtrade/freqtrade/worker.py", line 133, in _process
    state_changed = self.freqtrade.process()
  File "/root/freqtrade/freqtrade/freqtradebot.py", line 121, in process
    self.strategy.informative_pairs())
  File "/root/freqtrade/freqtrade/data/dataprovider.py", line 35, in refresh
    self._exchange.refresh_latest_ohlcv(pairlist)
  File "/root/freqtrade/freqtrade/exchange/exchange.py", line 544, in refresh_latest_ohlcv
    asyncio.gather(*input_coroutines, return_exceptions=True))
  File "/root/miniconda3/lib/python3.7/asyncio/base_events.py", line 571, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.
2019-04-25 12:09:30,233 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'type': status, 'status': 'process died'}
2019-04-25 12:09:30,349 - freqtrade.freqtradebot - INFO - Cleaning up modules ...
2019-04-25 12:09:30,350 - freqtrade.rpc.rpc_manager - INFO - Cleaning up rpc modules ...
(.env) (base) root@vmanager6003:~/freqtrade# 
xmatthias commented 5 years ago

you shouldn't collect backtesting data from the same ip/server than running a live/dry-run bot ... they'll always impact each other.

The logs indicate no DDOs protection though - but as if the API was down / the network to the server was temporarily down ... (ddos protection rasises a DDoSProtection error, not a RequestTimeout).

hroff-1902 commented 5 years ago

@Prossi79, could you pls attach your config? (be careful. remove exchange and telegram keys and passwords)

Prossi79 commented 5 years ago

@xmatthias .... mmmh, that is not very user-friendly :( I have just one Linux machine running, that is my VPS. I did not manage to run FT on my windows laptop.

Is there no way around this issue?

Prossi79 commented 5 years ago

@hroff-1902 here is my config:

`{ "max_open_trades": 3, "stake_currency": "BTC", "stake_amount": 0.1, "fiat_display_currency": "USD",

"process_only_new_candles": true,

"dry_run": false,
"db_url": "sqlite:///tradesv3.prodrun.sqlite",

"unfilledtimeout": {
    "buy": 10,
    "sell": 30
},

"bid_strategy": {
    "ask_last_balance": 0.75,
    "use_order_book": false,
    "order_book_top": 1,
    "check_depth_of_market": {
        "enabled": false,
        "bids_to_ask_delta": 1
    }
},

"ask_strategy":{
    "use_order_book": false,
    "order_book_min": 1,
    "order_book_max": 10
}, 

"order_types": {
"buy": "limit",
"sell": "limit",
"stoploss": "market",
"stoploss_on_exchange": false,
"stoploss_on_exchange_interval": 60
},

"exchange": {
    "name": "binance",
    "key": "xxxx",
    "secret": "xxxx",
    "ccxt_config": {"enableRateLimit": true},
    "ccxt_async_config": {
        "enableRateLimit": false
    },
    "pair_whitelist": [

        "ETH/BTC",
        "XRP/BTC",
        "LTC/BTC",
        "BCH/BTC",            
        "EOS/BTC",
        "BNB/BTC",
        "XLM/BTC",
        "ADA/BTC",        
        "TRX/BTC",
        "BSV/BTC",
        "XMR/BTC",   
        "DASH/BTC",
        "IOTA/BTC",
        "NEO/BTC",
        "ETC/BTC",            
        "ONT/BTC",
        "XEM/BTC",
        "ZEC/BTC",
        "VET/BTC",
        "BAT/BTC",
        "OMG/BTC",            
        "BTG/BTC",
        "QTUM/BTC",
        "WAVES/BTC",
        "REP/BTC",
        "DCR/BTC",
        "LSK/BTC",
        "RVN/BTC",
        "NANO/BTC",         
        "ZIL/BTC",
        "ZRX/BTC"

    ],
    "pair_blacklist": [

        "CRO/BTC",
        "DOGE/BTC",
        "XTC/BTC",          
        "MKR/BTC",
        "USDT/BTC"

    ]

}, 

"experimental": {
    "use_sell_signal": true,
    "sell_profit_only": true,
    "ignore_roi_if_buy_signal": false

},
"edge": {
    "enabled": false,
    "process_throttle_secs": 3600,
    "calculate_since_number_of_days": 10,
    "capital_available_percentage": 0.8,
    "allowed_risk": 0.01,
    "stoploss_range_min": -0.01,
    "stoploss_range_max": -0.05,
    "stoploss_range_step": -0.01,
    "minimum_winrate": 0.30,
    "minimum_expectancy": 0.10,
    "min_trade_number": 10,
    "max_trade_duration_minute": 1440,
    "remove_pumps": false
},
"telegram": {
    "enabled": true,
    "token": "xxx",
    "chat_id": "xxx"
},
"initial_state": "running",
"forcebuy_enable": false,
"internals": {
    "process_throttle_secs": 4
}

}`

xmatthias commented 5 years ago

well - you risk bans - that's nothing to do with user-friendly or not - it's the exchange that's blocking you, not us. Add in a high enough limit ("enableRateLimit": true, "rateLimit": 1000 in the ccxt_async_config section ) to avoid bans - but then downloading data will take forever since ccxt will pause 1 second between each subsequent call. If you only download (nothing else running on that box) - then going to 200 (0.2s) will work for binance (i tested that downloading all of last years 1m data ...)

Also when you backtest, you eat away the machines resources, so a live/dry-run bot could suffer from that as well. Me personally, i would not risk that - however it's your decision, not mine.

For windows installation, you should try WSL, that's a linux subsystem (ubuntu by default) - so the linux installation guide does work. I think it's been in since windows 8 in experimental mode - but with windows10 you can install it from the windows store...

Prossi79 commented 5 years ago

Thank you @xmatthias ! : ) it was not meant in a destructive way, but okay I understand the root cause issue now

xmatthias commented 5 years ago

didn't understand it that way - there's just nothing i would know off that we can do against that :disappointed:

xmatthias commented 5 years ago

I'll close this as this can be prevented using rateLimit, which is now also included in the sample settings.