Closed mostlyvirtual closed 4 years ago
Please check the logs for any message > INFO level which did happen in the timeframe (+-1-2 minutes) of this trade. Maybe some update or whatnot failed.
For the buy side: the Buy telegram message is sent when the buy order is created. Fees are not known at that point - so the buy message is working as intended (it's sending you all information known at that point in time).
for the fee handling itself - this is a known issue to some regard (#2649) - and especially applies when non-standard fees are applied. The issue itself proposes to move this to the users responsibility, which i disagree with - but i didn't yet have time to fully dig into this.
To help in aligning the fee handling between all exchanges, the following data would be helpful - as this should contain the information on fees applied.
import ccxt
from datetime import datetime
ct = ccxt.bittrex({'apiKey': '<key>', 'secret': '<secret>'}
ct.load_markets()
since = int(datetime(2020,4,1).timestamp() * 1000)
res= ct.fetch_my_trades('BTC/USD', since=since)
print(res)
Feel free to remove all trades which are not the above - i'll really just need one buy, and one sell trade (which ideally align).
Hello @xmatthias - thank you for your reply.
INFO messages before and after the trade look good to me:
2020-04-01 14:09:20,924 - freqtrade.worker - INFO - Bot heartbeat. PID=3386, version='2020.3', state='RUNNING'
2020-04-01 14:10:06,847 - freqtrade.data.converter - INFO - Missing data fillup for BTC/USD: before: 5759 - after: 5776
2020-04-01 14:10:11,969 - freqtrade.data.converter - INFO - Missing data fillup for BTC/USD: before: 5759 - after: 5776
2020-04-01 14:10:12,161 - freqtrade.wallets - INFO - Wallets synced.
2020-04-01 14:10:12,163 - freqtrade.freqtradebot - INFO - Buy signal found: about create a new trade with stake_amount: 40 ...
2020-04-01 14:10:12,167 - freqtrade.freqtradebot - INFO - Executing Buy for BTC/USD
2020-04-01 14:10:12,167 - freqtrade.freqtradebot - INFO - Using Last Bid / Last Price
2020-04-01 14:10:16,633 - freqtrade.wallets - INFO - Wallets synced.
2020-04-01 14:10:16,639 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'type': buy, 'exchange': 'Bittrex', 'pair': 'BTC/USD', 'limit': 6228.639, 'order_type': 'limit', 'stake_amount': 40.0, 'stake_currency': 'USD', 'fiat_currency': 'USD', 'amount': 0.006421948679318227, 'open_date': datetime.datetime(2020, 4, 1, 12, 10, 15, 150872), 'current_rate': 6228.639}
2020-04-01 14:10:18,358 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=22, pair=BTC/USD, amount=0.00642195, open_rate=6228.63900000, open_since=2020-04-01 12:10:15)
2020-04-01 14:10:23,139 - freqtrade.worker - INFO - Bot heartbeat. PID=3386, version='2020.3', state='RUNNING'
2020-04-01 14:10:23,378 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=22, pair=BTC/USD, amount=0.00642195, open_rate=6228.63900000, open_since=2020-04-01 12:10:15)
2020-04-01 14:10:28,381 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=22, pair=BTC/USD, amount=0.00642195, open_rate=6228.63900000, open_since=2020-04-01 12:10:15)
2020-04-01 14:10:33,368 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=22, pair=BTC/USD
From the sqlite db, this is the trade in question:
22|bittrex|BTC/USD|0|0.0025|0.0025|6228.639|6228.639|40.0999458045092|6242.397|6242.397|-0.00278972|-0.11186756|40.0|0.00642194|2020-04-01 12:10:15.150872|2020-04-01 14:31:49.893829||5917.20705|-0.05|5917.20705|-0.05||2020-04-01 12:13:01.312457|6254.363|6190.406|sell_signal|Strategy005|5
I just now noticed that the 5th and 6th columns are fee_open and fee_closed. Not sure what 0.0025 is, but that is wrong. If it's BTC, then that's $17, so clearly not correct, and I don't suspect it's BTC anyway. If it's USD, which is more likely, it should be 0.1%, which should be roughly 4cents.
Thanks for the explanation about the fee - that makes total sense now that I think of it. The bot cannot know what the fee was until the order has been filled, unless there was a way of setting 0.1% fee in the config.json file, but I guess that's what you meant when you said "to move this to the user's responsibility".
This is the return from the exchange for the ccxt command you asked:
{'id': '518083af-3419-473e-8601-af4cc41cd2fb', 'side': 'buy', 'order': '518083af-3419-473e-8601-af4cc41cd2fb', 'type': 'limit', 'price': 6228.638998495782, 'amount': 0.00642194, 'cost': 39.99994593, 'symbol': 'BTC/USD', 'timestamp': 1585743172950, 'datetime': '2020-04-01T12:12:52.950Z', 'fee': {'cost': 0.03999994, 'currency': 'USD'}, 'info': {'id': '518083af-3419-473e-8601-af4cc41cd2fb', 'timestamp': 1585743015120, 'datetime': '2020-04-01T12:10:15.120Z', 'lastTradeTimestamp': 1585743172950, 'symbol': 'BTC/USD', 'type': 'limit', 'side': 'buy', 'price': 6228.639, 'cost': 39.99994593, 'average': 6228.638998495782, 'amount': 0.00642194, 'filled': 0.00642194, 'remaining': 0.0, 'status': 'closed', 'fee': {'cost': 0.03999994, 'currency': 'USD'}, 'info': {'id': '518083af-3419-473e-8601-af4cc41cd2fb', 'marketSymbol': 'BTC-USD', 'direction': 'BUY', 'type': 'LIMIT', 'quantity': '0.00642194', 'limit': '6228.63900000', 'timeInForce': 'GOOD_TIL_CANCELLED', 'fillQuantity': '0.00642194', 'commission': '0.03999994', 'proceeds': '39.99994593', 'status': 'CLOSED', 'createdAt': '2020-04-01T12:10:15.12Z', 'updatedAt': '2020-04-01T12:12:52.95Z', 'closedAt': '2020-04-01T12:12:52.95Z'}}}, {'id': '7922eeb1-3faa-4024-8b6f-1126501907b3', 'side': 'sell', 'order': '7922eeb1-3faa-4024-8b6f-1126501907b3', 'type': 'limit', 'price': 6242.396999971971, 'amount': 0.00642194, 'cost': 40.08829899, 'symbol': 'BTC/USD', 'timestamp': 1585751503860, 'datetime': '2020-04-01T14:31:43.860Z', 'fee': {'cost': 0.04008829, 'currency': 'USD'}, 'info': {'id': '7922eeb1-3faa-4024-8b6f-1126501907b3', 'timestamp': 1585750509360, 'datetime': '2020-04-01T14:15:09.360Z', 'lastTradeTimestamp': 1585751503860, 'symbol': 'BTC/USD', 'type': 'limit', 'side': 'sell', 'price': 6242.397, 'cost': 40.08829899, 'average': 6242.396999971971, 'amount': 0.00642194, 'filled': 0.00642194, 'remaining': 0.0, 'status': 'closed', 'fee': {'cost': 0.04008829, 'currency': 'USD'}, 'info': {'id': '7922eeb1-3faa-4024-8b6f-1126501907b3', 'marketSymbol': 'BTC-USD', 'direction': 'SELL', 'type': 'LIMIT', 'quantity': '0.00642194', 'limit': '6242.39700000', 'timeInForce': 'GOOD_TIL_CANCELLED', 'fillQuantity': '0.00642194', 'commission': '0.04008829', 'proceeds': '40.08829899', 'status': 'CLOSED', 'createdAt': '2020-04-01T14:15:09.36Z', 'updatedAt': '2020-04-01T14:31:43.86Z', 'closedAt': '2020-04-01T14:31:43.86Z'}}}
This is the same trade as outlined in my original example, and the one that I extracted from the sqlite db - I used the same one to consistency.
I used a json to csv convertor to make this easier to read for me - not sure if it helps you, but I attached the file anyway. result.xlsx
fee_open and fee_closed are as ratio - so 0.0025 would be 0.25% ... (now the problem is - taken from which side ... and from which ... - therefore i'm currently gathering all possible returns from exchanges).
Json data is perfectly fine ... that's actually easier to automatically read than csv / xlsx - and i don't really need it "readable" - that's what the screenshots provide - i need to know how the actual result from the API looks like.
For Bittrex at least there is a discrepancy between the fees listed for markets and actual fees recorded. For instance, running ct.fetch_my_trades('BTC/USD' returns the following:
{"BTC/USD":{"percentage":true,"tierBased":false,"maker":0.0025,"taker":0.0025,"precision":{"amount":8,"price":3},"limits":{"amount":{"min":0.00025427,"max":"None"},"price":{"min":0.001,"max":"None"}},"id":"USD-BTC","symbol":"BTC/USD","base":"BTC","quote":"USD","baseId":"BTC","quoteId":"USD","active":true,"info":{"symbol":"BTC-USD","baseCurrencySymbol":"BTC","quoteCurrencySymbol":"USD","minTradeSize":"0.00025427","precision":3,"status":"ONLINE","createdAt":"2018-05-31T13:24:40.77Z","prohibitedIn":[]}}}
Which lists fees as 0.0025. Of course this can change based on volume or discounts, but with a recent trade I calculate a fee of 0.001 based on:
{"id":"XXX","side":"buy","order":"XXX","type":"limit","price":7083.515999929165,"amount":0.00282345,"cost":19.99995325,"symbol":"BTC/USD","timestamp":1587565352960,"datetime":"2020-04-22T14:22:32.960Z","fee":{"cost":0.01999995,"currency":"USD"},"info":{"id":"XXX","timestamp":1587565310790,"datetime":"2020-04-22T14:21:50.790Z","lastTradeTimestamp":1587565352960,"symbol":"BTC/USD","type":"limit","side":"buy","price":7083.516,"cost":19.99995325,"average":7083.515999929165,"amount":0.00282345,"filled":0.00282345,"remaining":0,"status":"closed","fee":{"cost":0.01999995,"currency":"USD"},"info":{"id":"XXX","marketSymbol":"BTC-USD","direction":"BUY","type":"LIMIT","quantity":"0.00282345","limit":"7083.51600000","timeInForce":"GOOD_TIL_CANCELLED","fillQuantity":"0.00282345","commission":"0.01999995","proceeds":"19.99995325","status":"CLOSED","createdAt":"2020-04-22T14:21:50.79Z","updatedAt":"2020-04-22T14:22:32.96Z","closedAt":"2020-04-22T14:22:32.96Z"}}}
Which is a purchase of 19.99995325 with 0.01999995 fee for a calculated 0.001.
The same is true for the sell side for the corresponding buy:
{"id":"XXX","side":"sell","order":"XXX","type":"limit","price":7120.421997910358,"amount":0.00282345,"cost":20.10415549,"symbol":"BTC/USD","timestamp":1587566339830,"datetime":"2020-04-22T14:38:59.830Z","fee":{"cost":0.02010415,"currency":"USD"},"info":{"id":"XXX","timestamp":1587566339830,"datetime":"2020-04-22T14:38:59.830Z","lastTradeTimestamp":1587566339830,"symbol":"BTC/USD","type":"limit","side":"sell","price":7120.42,"cost":20.10415549,"average":7120.421997910358,"amount":0.00282345,"filled":0.00282345,"remaining":0,"status":"closed","fee":{"cost":0.02010415,"currency":"USD"},"info":{"id":"XXX","marketSymbol":"BTC-USD","direction":"SELL","type":"LIMIT","quantity":"0.00282345","limit":"7120.42000000","timeInForce":"GOOD_TIL_CANCELLED","fillQuantity":"0.00282345","commission":"0.02010415","proceeds":"20.10415549","status":"CLOSED","createdAt":"2020-04-22T14:38:59.83Z","updatedAt":"2020-04-22T14:38:59.83Z","closedAt":"2020-04-22T14:38:59.83Z"}}}
Which is 20.10415549 sell for 0.02010415 fee to get 0.001 again. This later in inteface.stop_loss_reached() was calculated as a profit of -0.03642967. When freqtrade gets the order back from the exchange wouldn't it be better to extract and use the actual fees recorded so profit and stoploss calculations can be accurate?
Let me repeat what i said above: this is a known issue for the most part (#2649) - and i hope to have some time this weekend to finish the implementation for proper fee handling.
@xmatthias do you need me to test this? is it in the default develop branch?
no it's not, it's in the feat/fee_handling
branch.
I've been testing it myself and it seems ok so far - but my time for that is currently limited, so help in testing it is highly apreciated.
@xmatthias I started a bot now on that branch: 2020-05-08 22:01:20,135 - freqtrade.worker - INFO - Bot heartbeat. PID=4546, version='develop-e92d3867', state='RUNNING'
I'll let you know after it does a few trades, but it could be a few days.
Thanks.
the fix was merged into develop, pls verify
I only had one order since moving to develop-e92d3867 and it was closed correctly - i.e. all the amounts matched perfectly with my Bittrex panel.
67|bittrex|HBAR/USD|0|0.002|0.08000002|USD|0.002|0.08031528|USD|0.03535069|0.03535068|40.080011337606|0.03549|0.03549|-6.695e-05|-0.00268326|40.0|1131.51995944|2020-05-12 23:30:10.737901|2020-05-13 15:34:51.345353||0.0335831555|-0.05|0.0335831555|-0.05||2020-05-12 23:33:29.099709|0.03636306|0.03499998|sell_signal|Strategy005|5
I'll update this bot to develop branch and restart it. If you want, you can keep this issue closed for now, but I'll keep an eye out for future trades, until this makes it into master.
sure, thanks 👍
Step 1: Have you search for this issue before posting it?
Yes, I've searched for this issue, and also discussed it in Slack.
Step 2: Describe your environment
Step 3: Describe the problem:
The profit/loss is not accurate in the buy/sell messages, nor is it accurate then in /performance or /daily.
Steps to reproduce:
Observed Results:
When the coin was bought the bot said that it spent 40 USD, but Bittrex order history reports 40.03994587. Quite possible that the extra amount is the fee, but I think the bot should account for that if I understand correctly? When the coin was sold, the bot said that the trade lost money, but it actually made a profit, even including the fees.
I expected the buy/sell orders to account for the fees, and at the end to report accurate profit/loss percentage.
Relevant code exceptions or logs:
Scenario 1
Bot sends following Telegram message:
When checking Bittrex order history, we can see the 'Open Rate' and 'Amount' are accurate, but the 'Total' is not:
When it tried to close this trade, it sent the following Telegram message:
When checking Bittrex order history, we can see that it actually made a profit:
If we do 0.00642194 BTC x 6,242.39699997 USD = 40.08829898998734 USD If we do 40.08829898998734 USD (total money credited to account after sell) - 0.04008829 (sell commission) = 40.04821070 USD - 0.03999994 USD (commission on buy order) = 0.00826483 USD (our massive profit) which means that we made a profit of 0.02%, not a loss of -0.28%.
I've observed this with other orders, as it can be seen clearly in the order history window, especially when sell order is right after buy order.
In the next screenshot, the bot reported a loss of 0.33%, but the loss was really 1.66%:
In the next screenshot, the bot reported a profit of 0.17%, but in fact it was 0.49%
Finally, in closing, I want to mention that it happens with other pairs as well, not just USD/BTC.