freqtrade / freqtrade

Free, open source crypto trading bot
https://www.freqtrade.io
GNU General Public License v3.0
27.03k stars 5.86k forks source link

Candles not updating #9376

Closed bonze82 closed 8 months ago

bonze82 commented 9 months ago

Describe your environment

Note: All issues other than enhancement requests will be closed without further comment if the above template is deleted or not filled out.

Describe the problem:

current trade value never changes.

Steps to reproduce:

  1. Run live
  2. Have an open trade on binanceus
  3. Tried forcing the trade current value to update twice with no result.

Observed Results:

Open trade current value never changes. Expected current value to change with market value allowing dashboard to update.

Relevant code exceptions or logs

Note: Please copy/paste text of the messages, no screenshots of logs please.

2023-11-03 20:26:09 - freqtrade.wallets - INFO - Wallets synced.
2023-11-03 20:26:09 - freqtrade.freqtradebot - INFO - Long signal found: about create a new trade for MXC/USDT with stake_amount: 45.874414865069994 ...
2023-11-03 20:26:09 - freqtrade.freqtradebot - INFO - Order 5757767 was created for MXC/USDT and status is open.
2023-11-03 20:26:09 - freqtrade.wallets - INFO - Wallets synced.
2023-11-03 20:26:09 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'trade_id': 2, 'type': entry, 'buy_tag': 'ewo2', 'enter_tag': 'ewo2', 'exchange': 'Binanceus', 'pair': 'MXC/USDT', 'leverage': 1.0, 'direction': 'Long', 'limit': 0.00627, 'open_rate': 0.00627, 'order_type': 'limit', 'stake_amount': 45.874414865069994, 'stake_currency': 'USDT', 'base_currency': 'MXC', 'fiat_currency': 'USD', 'amount': 7316.0, 'open_date': datetime.datetime(2023, 11, 3, 20, 26, 9, 686964, tzinfo=datetime.timezone.utc), 'current_rate': 0.00627, 'sub_trade': False}
2023-11-03 20:26:09 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring HNT/USDT from whitelist. Market is not active.
2023-11-03 20:26:09 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring BUSD/USDT from whitelist. Market is not active.
2023-11-03 20:26:09 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring OMG/USDT from whitelist. Market is not active.
2023-11-03 20:26:09 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring POLY/USDT from whitelist. Market is not active.
2023-11-03 20:26:09 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring TRX/USDT from whitelist. Market is not active.
2023-11-03 20:26:09 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring UST/USDT from whitelist. Market is not active.
2023-11-03 20:26:09 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring SPELL/USDT from whitelist. Market is not active.
2023-11-03 20:26:09 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring JASMY/USDT from whitelist. Market is not active.
2023-11-03 20:26:09 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring SRM/USDT from whitelist. Market is not active.
2023-11-03 20:26:09 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring FLOKI8/USDT from whitelist. Market is not active.
2023-11-03 20:26:10 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=2, pair=MXC/USDT, amount=7316.49359889, is_short=False, leverage=1.0, open_rate=0.00627000, open_since=2023-11-03 20:26:09)
2023-11-03 20:26:14 - freqtrade.worker - INFO - Bot heartbeat. PID=1, version='2023.10', state='RUNNING'
~~~~
2023-11-03 20:27:54 - freqtrade.wallets - INFO - Wallets synced.
2023-11-03 20:27:54 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'trade_id': 2, 'type': entry_fill, 'buy_tag': 'ewo2', 'enter_tag': 'ewo2', 'exchange': 'Binanceus', 'pair': 'MXC/USDT', 'leverage': 1.0, 'direction': 'Long', 'limit': 0.00627, 'open_rate': 0.00627, 'order_type': 'limit', 'stake_amount': 45.68783472, 'stake_currency': 'USDT', 'base_currency': 'MXC', 'fiat_currency': 'USD', 'amount': 7286.736, 'open_date': datetime.datetime(2023, 11, 3, 20, 26, 9, 686964, tzinfo=datetime.timezone.utc), 'current_rate': 0.00627, 'sub_trade': True}
2023-11-03 20:28:19 - freqtrade.worker - INFO - Bot heartbeat. PID=1, version='2023.10', state='RUNNING'
2023-11-03 21:56:14 - freqtrade.plugins.pairlistmanager - WARNING - Pair BNB/USDT in your blacklist. Removing it from whitelist...
2023-11-03 21:56:14 - freqtrade.plugins.pairlistmanager - WARNING - Pair JAM/USDT in your blacklist. Removing it from whitelist...
2023-11-03 21:56:14 - freqtrade.plugins.pairlistmanager - INFO - Whitelist with 155 pairs: ['BTC/USDT', 'ETH/USDT', 'XRP/USDT', 'BCH/USDT', 'LTC/USDT', 'ADA/USDT', 'BAT/USDT', 'ETC/USDT', 'XLM/USDT', 'ZRX/USDT', 'DOGE/USDT', 'ATOM/USDT', 'NEO/USDT', 'VET/USDT', 'QTUM/USDT', 'ONT/USDT', 'KNC/USDT', 'VTHO/USDT', 'COMP/USDT', 'MKR/USDT', 'ONE/USDT', 'BAND/USDT', 'STORJ/USDT', 'UNI/USDT', 'SOL/USDT', 'EGLD/USDT', 'PAXG/USDT', 'OXT/USDT', 'ZEN/USDT', 'FIL/USDT', 'AAVE/USDT', 'GRT/USDT', 'SHIB/USDT', 'CRV/USDT', 'AXS/USDT', 'AVAX/USDT', 'CTSI/USDT', 'DOT/USDT', 'YFI/USDT', '1INCH/USDT', 'FTM/USDT', 'USDC/USDT', 'MATIC/USDT', 'MANA/USDT', 'ALGO/USDT', 'LINK/USDT', 'EOS/USDT', 'ZEC/USDT', 'ENJ/USDT', 'NEAR/USDT', 'SUSHI/USDT', 'LRC/USDT', 'LPT/USDT', 'NMR/USDT', 'SLP/USDT', 'ANT/USDT', 'CHZ/USDT', 'OGN/USDT', 'GALA/USDT', 'TLM/USDT', 'SNX/USDT', 'AUDIO/USDT', 'ENS/USDT', 'REQ/USDT', 'APE/USDT', 'FLUX/USDT', 'COTI/USDT', 'VOXEL/USDT', 'RLC/USDT', 'BICO/USDT', 'API3/USDT', 'BNT/USDT', 'IMX/USDT', 'FLOW/USDT', 'GTC/USDT', 'THETA/USDT', 'TFUEL/USDT', 'OCEAN/USDT', 'LAZIO/USDT', 'SANTOS/USDT', 'ALPINE/USDT', 'PORTO/USDT', 'REN/USDT', 'CELR/USDT', 'SKL/USDT', 'VITE/USDT', 'WAXP/USDT', 'LTO/USDT', 'FET/USDT', 'BOND/USDT', 'LOKA/USDT', 'ICP/USDT', 'T/USDT', 'OP/USDT', 'ROSE/USDT', 'CELO/USDT', 'KDA/USDT', 'KSM/USDT', 'ACH/USDT', 'DAR/USDT', 'RNDR/USDT', 'SYS/USDT', 'RAD/USDT', 'ILV/USDT', 'LDO/USDT', 'RARE/USDT', 'LSK/USDT', 'DGB/USDT', 'REEF/USDT', 'ALICE/USDT', 'FORTH/USDT', 'ASTR/USDT', 'BTRST/USDT', 'GAL/USDT', 'SAND/USDT', 'BAL/USDT', 'GLM/USDT', 'CLV/USDT', 'TUSD/USDT', 'QNT/USDT', 'STG/USDT', 'AXL/USDT', 'KAVA/USDT', 'APT/USDT', 'MASK/USDT', 'BOSON/USDT', 'POND/USDT', 'MXC/USDT', 'TRAC/USDT', 'PROM/USDT', 'DIA/USDT', 'LOOM/USDT', 'STMX/USDT', 'POLYX/USDT', 'IOST/USDT', 'ARB/USDT', 'FLOKI/USDT', 'XEC/USDT', 'BLUR/USDT', 'ANKR/USDT', 'DAI/USDT', 'DASH/USDT', 'HBAR/USDT', 'ICX/USDT', 'IOTA/USDT', 'RVN/USDT', 'WAVES/USDT', 'XNO/USDT', 'XTZ/USDT', 'ZIL/USDT', 'ORBS/USDT', 'CUDOS/USDT', 'ADX/USDT', 'FORT/USDT', 'SUI/USDT']
2023-11-03 21:56:18 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring HNT/USDT from whitelist. Market is not active.
2023-11-03 21:56:18 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring BUSD/USDT from whitelist. Market is not active.
2023-11-03 21:56:18 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring OMG/USDT from whitelist. Market is not active.
2023-11-03 21:56:18 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring POLY/USDT from whitelist. Market is not active.
2023-11-03 21:56:18 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring TRX/USDT from whitelist. Market is not active.
2023-11-03 21:56:18 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring UST/USDT from whitelist. Market is not active.
2023-11-03 21:56:18 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring SPELL/USDT from whitelist. Market is not active.
2023-11-03 21:56:18 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring JASMY/USDT from whitelist. Market is not active.
2023-11-03 21:56:18 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring SRM/USDT from whitelist. Market is not active.
2023-11-03 21:56:18 - freqtrade.plugins.pairlist.IPairList - INFO - Ignoring FLOKI8/USDT from whitelist. Market is not active.
2023-11-03 22:53:47 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=2, pair=MXC/USDT, amount=7286.00000000, is_short=False, leverage=1.0, open_rate=0.00627000, open_since=2023-11-03 20:26:09)
2023-11-03 22:53:47 - freqtrade.persistence.trade_model - INFO - Updating trade (id=2) ...
2023-11-03 22:53:47 - freqtrade.persistence.trade_model - INFO - LIMIT_BUY has been fulfilled for Trade(id=2, pair=MXC/USDT, amount=7286.73600000, is_short=False, leverage=1.0, open_rate=0.00627000, open_since=2023-11-03 20:26:09).
2023-11-03 22:53:47 - freqtrade.wallets - INFO - Wallets synced.
2023-11-03 22:53:47 - freqtrade.freqtradebot - INFO - handled order 5757767
2023-11-03 22:53:54 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=2, pair=MXC/USDT, amount=7286.00000000, is_short=False, leverage=1.0, open_rate=0.00627000, open_since=2023-11-03 20:26:09)
2023-11-03 22:53:54 - freqtrade.persistence.trade_model - INFO - Updating trade (id=2) ...
2023-11-03 22:53:54 - freqtrade.persistence.trade_model - INFO - LIMIT_BUY has been fulfilled for Trade(id=2, pair=MXC/USDT, amount=7286.73600000, is_short=False, leverage=1.0, open_rate=0.00627000, open_since=2023-11-03 20:26:09).
2023-11-03 22:53:54 - freqtrade.wallets - INFO - Wallets synced.
2023-11-03 22:53:54 - freqtrade.freqtradebot - INFO - handled order 5757767
2023-11-03 22:53:59 - freqtrade.worker - INFO - Bot heartbeat. PID=1, version='2023.10', state='RUNNING'
2023-11-03 23:07:29 - freqtrade.worker - INFO - Bot heartbeat. PID=1, version='2023.10', state='RUNNING'
stash86 commented 9 months ago

First, please update to latest stable. We won't help debug issues on old version

Second, I don't see the problem in the log.

bonze82 commented 9 months ago

Ok updated to dev freqtrade docker-2023.11-dev-2a0175f6

For the second I see the bot is updating value for the trade, evident under trade, but never updates under the dashboard to reflect newer prices.

bonze82 commented 9 months ago

Ok I figured it out the bot waited for the coin to drop BACK down from .00659 to .0063, not a lot of money but when your talking about 10k coins it boils down to a difference between making a profit and loosing a profit.

xmatthias commented 9 months ago

well for now, this kindof sounds like it's a topic about your strategy / configuration, rather than a problem with the bot itself.

The bot created a long trade - which filled about half an hour later (once price dropped down to the price you defined as your entry price).

This will correspond to your pricing settings - which is a section of the documentation you might want to read carefully - and with slight prior market understanding, will help you understand why your trade filled later than you expected. a live trade placed at that price would've behaved similarily / identically - and it's all based on what price you define (best ask price, best bid price, ...).

bonze82 commented 8 months ago

This was an exit problem. The bot exited 5 hours after the price increased. The bot exited after the price fell to just above the initial entry.

Entry at .00627 -> went down to .00626 -> after an hour goes to .00659 for 5 hours -> back down to .0063 which the bot sold as a trailing stop loss. I figured when it leveled off for 5 hours the bot would sell the stock as a roi.

stash86 commented 8 months ago

Well, without looking at any code and full logs, we won't know what exactly happened there.

xmatthias commented 8 months ago

This was an exit problem.

The logs only show a (delayed) entry - which is what the answer is based on.

bonze82 commented 8 months ago

bug.txt startup.txt

entire log available upon request. the entire log is 3mb in size.

xmatthias commented 8 months ago

Well what makes you belive it should be selling earlier / at a different rate?

looking at your startup logs - you're using minimal ROI and no trailing stoploss (for simple reference / potential future reference).

minimal_roi = {'0': 0.16899999999999998, '15': 0.086, '65': 0.038, '136': 0`


the bug.txt log is however roughly the same log as in the initial message (though complete, without interruption in the middle). It doesn't change the picture though - as it ends 2 heartbeats after the "entry_fill" message has been sent. Between these - the entry order was open - from 20:26:06 to 20:27:54 - where the order filled.

Until that point you didn't actually own the coin, but you had an open limit order (in regular words - the intend to buy a coin at a price below the best ask price). Obviously (as you didn't own the coin yet) - there's also no reason to even check for exits (which the bot didn't until the order filled).

There's no exit in the logs - therefore we also can't really look at any exit (logs start once the entry is complete).

Assuming this is where you suspect the "problem" - then the above recommendation (https://github.com/freqtrade/freqtrade/issues/9376#issuecomment-1793373525) remains - please familiarize yourself with how pricing works.

bonze82 commented 8 months ago

Ok so I familiarized myself with pricing a bit better and minimum roi, even went a bit further and dug deeper into the trade to make sure I wasn't missing something. The trade details exposes the bug in greater depth. the trade was bought at .00627, fell to .00626 and 5 minutes later went up to .00659 for 5 hours. After 5 hours the stock fell back down to .00630 and on the way down the bot sold at .0063. My initial question and bug report stands why if my roi was at 5.1% for 300 minutes and the min roi is set to 65 minutes for over 3.8% did the bot not sell the trade.

Btw the bot reported the value as .00626 for those 300 minutes until the trade started falling and then sold the trade for a gain of .47% which resulted in a loss. Why is the bot not getting accurate numbers from the exchange part of freqtrade to the bot part of freqtrade. The trade tab showed the exchange reporting valid numbers from the exchange yet the dashboard was reporting old information for 5 hours. I'm not great with code but I am willing to dig into the issue if pointed to the right area.

stash86 commented 8 months ago

if you can open the database, or can access the API, give us the orders detail for that trade.

bonze82 commented 8 months ago

bug complete.txt Complete bug log sorry I thought the prior was complete.

Still reasoning stands why did it take 5 hours to sell instead of selling after the 65 minute roi of 4.7% which is over the 3.8%. In my mind it should have exited 95 minutes after entry. I can understand if it was waiting to see if it would climb up higher, and that leaves me with the question what is the standard long wait to sell time. Or is my thought process severely skewed in some way.

stash86 commented 8 months ago

you will need to give us the database or the detailed trade response from API.

despite the entry_fill notif at 2023-11-03 20:27:54,387, you have another entry filled at 2023-11-03 22:53:47,526, and I believe that's the real limit entry filled, at least according to the bot. Only then, the bot will check for exit

bonze82 commented 8 months ago

Screenshot_2023-11-07_19-33-59 I think this is what your asking for, this is a screenshot of the trade on the exchange. I'm not sure how to export the database so sending them as a zip database files.zip .

stash86 commented 8 months ago

this is what happen when you are trading low volume pair

The time of your entry image

The next candle that has significant volume. This is where your bot initially try to exit, if you see the orders, but only got partially filled (21 out of 7286). The next candle is where the 7265 filled, which is far below the previous candle. image

It's not a FT bug. Learn about volume pairlist and why it's dangerous to trade low volume coin.

stash86 commented 8 months ago

If you look at freqtrade, there are plenty of gaps on timestamp of the candles, which means there were plenty of 0 volume candles. If you look at the chart on your binanceus app, you will see those long no-activity chart. Learn to look at the real-time chart and what it actually does, and don't only focus on the close price

bonze82 commented 8 months ago

What can I do to avoid trading low volume pair?

xmatthias commented 8 months ago

well let's summarize what i found in the logs (just for MXC): All activity happening between 2023-11-03 20:26 and 2023-11-04 03:16 MXC trade opened at 20:26:09 MXC order filled at 20:27:54

MXC detected and placed a ROI exit at 02:45 MXC Limit sell order filled partially at 03:15, a new exit was placed as trailing stoploss - which was denied by the strategy 4 times within that minute before being allowed through - and filled at 03:15:49.

If we look at this in an image (30m, binanceus doesn't seem to provide more history through their UI), it becomes clear why the orders didn't fill - this coin has 0 volume.

image

open_rate = 0.00627 close_rate= 0.00656 amount=7265.0 (in MXC)

Based on the chart, these values also seem reasonable / correct (both as open and as high value).


Now you can't take these values and say "my profit was open / close" - that's the coin movement - which does NOT equate to your actual profit - which will have to deduct fees - based on the database, 0.4% and 0.6% on open / close respectively.

Freqtrade accounts for these fees at each step - so the ROI / stop you see will already have fees included (with the assumption that exit fee == entry fee - which may or may not hold true).

in theory, the above would get you to a stake-amount of 45.5515 USDT. the trade value however is (open_rate * amount * (1-fee)) - so 45.3693438 (you already paid almost 2c in fees).

Obviously, the same goes on the high point - `0.00656 7265.0 (1 - 0.006) - or 47.3724496. now doing the profit on the actual values, it's barely scratching at the high mark of 3.8% image

The trade also registered the ROI exit - which timed out after half an hour (according to the configuration) and was half-filled (as said above - pricing...). Timing the order out earlier might result in a new exit at a higher price - but maybe that also doesn't fill.

The initial exit filled for 21 MXC - which left the big part fo the position to be at the stop level - which then obviously dragged the trade into a loss.

The bot did work as expected based on the available info.


image

Different pricing MAY have won this trade - but given the wide spread of this coin, might've also resulted in the trade opening WAY higher - and resulting in a stop way earlier. It's a tradeof, which - especially on very low volume coins like this one - there's often no correct answer.


You can use different pairlists, and either shorten your Volumepairlist length - or manually pick low volume pairs. you could also add a guard in your strategy (e.g. a rolling average over volume and comparing this to a shorter rolling average of the volume ...)

stash86 commented 8 months ago

What can I do to avoid trading low volume pair?

What I usually do is have this

dataframe['live_data_ok'] = (dataframe['volume'].rolling(window=72, min_periods=72).min() > 0)

What it does is it make sure that the past 72 candles has no 0 volume candle(s). You can change the length of check if you want to. There is no science in me choosing the number 72

bonze82 commented 8 months ago

Ok, thanks to both of you, I'm used to trading in the stock market and didn't realize trading crypto would be so different. One last question to stash86, I suck at python I understand the line of code above just not sure where it needs to go. I'm thinking the strategy but not sure where in the strategy or I might be completely wrong.

stash86 commented 8 months ago

Example

https://github.com/stash86/MultiMA_TSL/blob/main/user_data/strategies/Cenderawasih/Cenderawasih_3.py