search

freqtrade / freqtrade

Free, open source crypto trading bot
https://www.freqtrade.io
GNU General Public License v3.0
26.34k stars 5.78k forks source link

OKX leverage tier loading fails #8512

Closed tomjrtsmith closed 1 year ago

tomjrtsmith commented 1 year ago

Describe your environment

Note: All issues other than enhancement requests will be closed without further comment if the above template is deleted or not filled out.

Describe the problem:

Explain the problem you have encountered

Volume pairlist config for OKX

Steps to reproduce:

 "pairlists": [
    {
        "method": "VolumePairList",
        "number_assets": 100,
        "sort_key": "quoteVolume"
    },
    {"method": "AgeFilter", "min_days_listed": 30},
    {"method": "PriceFilter", "low_price_ratio": 0.01},
    {"method": "SpreadFilter", "max_spread_ratio": 0.005}
    ],

Observed Results:

Failing with this error, I've cut it down significantly from Binance config where it was asking to use candles as the config wasn't suitable, now this error

Relevant code exceptions or logs

Apr 17 06:57:14 instance-20221214-0105 freqtrade.exchange.exchange - INFO - Initializing leverage_tiers for 127 markets. This will take about a minute. Apr 17 06:57:14 instance-20221214-0105 freqtrade_ft5a[1458907]: 2023-04-17 06:57:14,273 - freqtrade.exchange.common - WARNING - get_market_leverage_tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest. Message: okx {"code":"51000","data":[],"msg":"Parameter uly error"}". Retrying still for 4 times. Apr 17 06:57:14 instance-20221214-0105 freqtrade.exchange.common - WARNING - get_market_leverage_tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest. Message: okx {"code":"51000","data":[],"msg":"Parameter uly error"}". Retrying still for 4 times. Apr 17 06:57:14 instance-20221214-0105 freqtrade_ft5a[1458907]: 2023-04-17 06:57:14,621 - freqtrade.exchange.common - WARNING - get_market_leverage_tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest. Message: okx {"code":"51000","data":[],"msg":"Parameter uly error"}". Retrying still for 3 times. Apr 17 06:57:14 instance-20221214-0105 freqtrade.exchange.common - WARNING - get_market_leverage_tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest. Message: okx {"code":"51000","data":[],"msg":"Parameter uly error"}". Retrying still for 3 times. Apr 17 06:57:14 instance-20221214-0105 freqtrade_ft5a[1458907]: 2023-04-17 06:57:14,966 - freqtrade.exchange.common - WARNING - get_market_leverage_tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest. Message: okx {"code":"51000","data":[],"msg":"Parameter uly error"}". Retrying still for 2 times. Apr 17 06:57:14 instance-20221214-0105 freqtrade.exchange.common - WARNING - get_market_leverage_tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest. Message: okx {"code":"51000","data":[],"msg":"Parameter uly error"}". Retrying still for 2 times. Apr 17 06:57:15 instance-20221214-0105 freqtrade_ft5a[1458907]: 2023-04-17 06:57:15,337 - freqtrade.exchange.common - WARNING - get_market_leverage_tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest. Message: okx {"code":"51000","data":[],"msg":"Parameter uly error"}". Retrying still for 1 times. Apr 17 06:57:15 instance-20221214-0105 freqtrade.exchange.common - WARNING - get_market_leverage_tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest. Message: okx {"code":"51000","data":[],"msg":"Parameter uly error"}". Retrying still for 1 times. Apr 17 06:57:15 instance-20221214-0105 freqtrade_ft5a[1458907]: 2023-04-17 06:57:15,694 - freqtrade.exchange.common - WARNING - get_market_leverage_tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest. Message: okx {"code":"51000","data":[],"msg":"Parameter uly error"}". Giving up. Apr 17 06:57:15 instance-20221214-0105 freqtrade.exchange.common - WARNING - get_market_leverage_tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest. Message: okx {"code":"51000","data":[],"msg":"Parameter uly error"}". Giving up. Apr 17 06:57:15 instance-20221214-0105 freqtrade_ft5a[1458907]: 2023-04-17 06:57:15,697 - freqtrade.commands.trade_commands - ERROR - cannot unpack non-iterable TemporaryError object Apr 17 06:57:15 instance-20221214-0105 freqtrade.commands.trade_commands - ERROR - cannot unpack non-iterable TemporaryError object

  // paste your log here
zhuque commented 1 year ago

met it too đź’”

roozich commented 1 year ago

INFO - Initializing leverage_tiers for 127 markets. WARNING - get_market_leverage_tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest.

GLMR added recently

roozich commented 1 year ago

@xmatthias how to handle leverage tiers newly added pairs on the futures market when using dynamic pair list ( volumePairList ) and prevent the bot from stoping.

xmatthias commented 1 year ago

GLMR is NOT a futures pair at this moment. CI shows the same errors since about an hour ago.

it seems like OKX messed up their API. best open a support ticket with them.

On a side node, and regarding the version posted in the original issue: We do only support the latest release, as well as the latest develop version, with uptodate dependencies. Your's is clearly not - so the first approach will have to be to update. we'll not be maintaining 2023.1 (your version) should changes on our end be required.

raoulbuzziol commented 1 year ago

Can the effect of the API glitch be mitigated by "just" skipping problematic pair? I have dozens open positions (in dry_run) and this morning all the bots completely stopped working because of a single pair (that I don't even use). I think this could be frequent "normal" issue.

The stacktrace is all inside freqtrade, so maybe it can be made a bit more resilient:

2023-04-17 09:50:05,518 - freqtrade.commands.trade_commands - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/freqtrade/freqtrade/commands/trade_commands.py", line 24, in start_trading
    worker = Worker(args)
  File "/freqtrade/freqtrade/worker.py", line 37, in __init__
    self._init(False)
  File "/freqtrade/freqtrade/worker.py", line 53, in _init
    self.freqtrade = FreqtradeBot(self._config)
  File "/freqtrade/freqtrade/freqtradebot.py", line 70, in __init__
    self.exchange = ExchangeResolver.load_exchange(
  File "/freqtrade/freqtrade/resolvers/exchange_resolver.py", line 34, in load_exchange
    exchange = ExchangeResolver._load_exchange(
  File "/freqtrade/freqtrade/resolvers/exchange_resolver.py", line 60, in _load_exchange
    exchange = ex_class(**kwargs)
  File "/freqtrade/freqtrade/exchange/exchange.py", line 195, in __init__
    self.fill_leverage_tiers()
  File "/freqtrade/freqtrade/exchange/exchange.py", line 2446, in fill_leverage_tiers
    leverage_tiers = self.load_leverage_tiers()
  File "/freqtrade/freqtrade/exchange/exchange.py", line 2405, in load_leverage_tiers
    for symbol, res in results:
TypeError: cannot unpack non-iterable TemporaryError object
amirhosein-fasihi commented 1 year ago

same issue

TonyStarks22 commented 1 year ago

change trading mode from spot to FUTURES.. Should have been fixed, if you are using Static pair mode

Converts the interval provided in minutes in config to seconds

    self.markets_refresh_interval: int = exchange_config.get(
        "markets_refresh_interval", 60) * 60

    if self.trading_mode != TradingMode.FUTURES and load_leverage_tiers:
        self.fill_leverage_tiers()
    self.additional_exchange_init()
xmatthias commented 1 year ago

@TonyStarks22 please don't post missleading "fixes" without understanding the implication.

While your change will probably disable leverage tier loading - that'll also implicitly mean that the bot can no longer open any position in futures mode - so while startup works - you've broken the bot. I've therefore hidden your comment (unfortunately there's no real fitting reason ...).

TonyStarks22 commented 1 year ago

I dont know, why you using the word mis leading, as im sharing the changes that made my bots working again and get rid of the error. It works for me, and i hope your is working now too. C @TonyStarks22 please don't post missleading "fixes" without understanding the implication. While your change will probably disable leverage tier loading - that'll also implicitly mean that the bot can no longer open any position in futures mode - so while startup works - you've broken the bot. I've therefore hidden your comment (unfortunately there's no real fitting reason ...).

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

xmatthias commented 1 year ago

@TonyStarks22 because your "fix" means that no leverage tiers are available.

while the bot will START - it will NOT be able to execute trades (at least not properly) - as the leverage tiers will not contain pairs - leading to errors once your bot is actually in positions - and either cause failures at that point, or cause wrong calculations for trades done with this modification. it's therefore misleading to users who would attempt to use your modification - as it's wrong and leading to unintended side-effects later on that you don't mention.

raoulbuzziol commented 1 year ago

Occasionally the bot dies with the below error, I think it's releated to this issue. I'm using develop docker image since yesterday, before I was using stable, therefore I don't know if the issue is new or not.

2023-04-18 03:46:59,182 - freqtrade.commands.trade_commands - ERROR - Parse error at offset 1507328: Missing a closing quotation mark in string.
2023-04-18 03:46:59,183 - freqtrade.commands.trade_commands - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/freqtrade/freqtrade/commands/trade_commands.py", line 25, in start_trading
    worker.run()
  File "/freqtrade/freqtrade/worker.py", line 75, in run
    state = self._worker(old_state=state)
  File "/freqtrade/freqtrade/worker.py", line 116, in _worker
    self._throttle(func=self._process_running, throttle_secs=self._throttle_secs,
  File "/freqtrade/freqtrade/worker.py", line 147, in _throttle
    result = func(*args, **kwargs)
  File "/freqtrade/freqtrade/worker.py", line 180, in _process_running
    self.freqtrade.process()
  File "/freqtrade/freqtrade/freqtradebot.py", line 207, in process
    self.exchange.reload_markets()
  File "/freqtrade/freqtrade/exchange/exchange.py", line 509, in reload_markets
    self.fill_leverage_tiers()
  File "/freqtrade/freqtrade/exchange/exchange.py", line 2449, in fill_leverage_tiers
    leverage_tiers = self.load_leverage_tiers()
  File "/freqtrade/freqtrade/exchange/exchange.py", line 2383, in load_leverage_tiers
    tiers_cached = self.load_cached_leverage_tiers(self._config['stake_currency'])
  File "/freqtrade/freqtrade/exchange/exchange.py", line 2434, in load_cached_leverage_tiers
    tiers = file_load_json(filename)
  File "/freqtrade/freqtrade/misc.py", line 128, in file_load_json
    pairdata = json_load(datafile)
  File "/freqtrade/freqtrade/misc.py", line 111, in json_load
    return rapidjson.load(datafile, number_mode=rapidjson.NM_NATIVE)
rapidjson.JSONDecodeError: Parse error at offset 1507328: Missing a closing quotation mark in string.
2023-04-18 03:46:59,227 - freqtrade.commands.trade_commands - INFO - worker found ... calling exit
2023-04-18 03:46:59,228 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'type': status, 'status': 'process died'}
2023-04-18 03:47:01,279 - freqtrade.freqtradebot - INFO - Cleaning up modules ...
2023-04-18 03:47:01,286 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'type': warning, 'status': "6 open trades active.\n\nHandle these trades manually on OKX, or '/start' the bot again and use '/stopentry' to handle open trades gracefully. \nNote: Trades are simulated (dry run)."}
2023-04-18 03:47:01,448 - freqtrade.rpc.rpc_manager - INFO - Cleaning up rpc modules ...
2023-04-18 03:47:01,449 - freqtrade.rpc.rpc_manager - INFO - Cleaning up rpc.telegram ...
2023-04-18 03:47:01,450 - apscheduler.scheduler - INFO - Scheduler has been shut down
xmatthias commented 1 year ago

@raoulbuzziol The error suggests that the cache file was written to disk only partially - and then fails to read the file.

While it shouldn't crash it - it's really suggesting a different error - most likely a "disk was full" error in a prior run - causing this issue (if the disk was full / near full, then this error will repeat itself with the next run). You should be able to work around this for now by simply deleting the cache file (something around user_data/data/okx/leverage_tiers* ....).

raoulbuzziol commented 1 year ago

@xmatthias I'm running multiple bots in the same place sharing the same user_data dir. So it's probably a race condition on that file. Just restarting the bot works, probably another bot succeded writing it correctly winning the race.

In my case mounting user_data/data folder in a different volume would probably solve the problem.

It never happened on stable branch. Since yesterday I'm running develop branch, and it happened 5 times, therefore I thought it was correlated to this issue.

xmatthias commented 1 year ago

not really - the file is usually cached for several days - so unless you always start all bots at exactly the same time (and they all take exactly the same time to load the tiers) - it's an unlikely issue.

Longjef commented 1 year ago

The problem you are talking about is not the bug problem mentioned in the title, is it. I also encountered the issue raised by the same title: WARNING - get market leverage_ tiers() returned exception: "Could not load leverage tiers for GLMR/USDT:USDT due to BadRequest。Can you first address the issue raised by the title

raoulbuzziol commented 1 year ago

@Longjef The issue from the title has been solved and in fact this issue is closed. You can find the fix in the develop branch and it's working fine.

Other [off] topics are my fault, I thought they where correlated, but they are not. I switched from stable to develop branch and found some instabilities. Might be better located in another issue.