freqtrade / freqtrade

Free, open source crypto trading bot
https://www.freqtrade.io
GNU General Public License v3.0
28.13k stars 6.03k forks source link

Profit/Loss in Bittrex order history doesn't match ft db/reports. #3131

Closed mostlyvirtual closed 4 years ago

mostlyvirtual commented 4 years ago

Step 1: Have you search for this issue before posting it?

Yes, I've searched for this issue, and also discussed it in Slack.

Step 2: Describe your environment

Step 3: Describe the problem:

The profit/loss is not accurate in the buy/sell messages, nor is it accurate then in /performance or /daily.

Steps to reproduce:

  1. Buy BTC/USD on Bittrex.
  2. Sell BTC/USD on Bittrex.
  3. Compare with order history in Bittrex.

Observed Results:

When the coin was bought the bot said that it spent 40 USD, but Bittrex order history reports 40.03994587. Quite possible that the extra amount is the fee, but I think the bot should account for that if I understand correctly? When the coin was sold, the bot said that the trade lost money, but it actually made a profit, even including the fees.

I expected the buy/sell orders to account for the fees, and at the end to report accurate profit/loss percentage.

Relevant code exceptions or logs:

Scenario 1

Bot sends following Telegram message:

Bittrex: Buying BTC/USD
Amount: 0.00642195
Open Rate: 6228.63900000
Current Rate: 6228.63900000
Total: (40.000000 USD, 40.000 USD)

When checking Bittrex order history, we can see the 'Open Rate' and 'Amount' are accurate, but the 'Total' is not:

Closed Date:
2020/04/01 15:12:52
Opened Date:
2020/04/01 15:10:15
Market:
USD-BTC
Side:
BUY
Type:
Limit
Price:
6,228.63900000 USD
Quantity Filled:
0.00642194 BTC
Total Quantity:
0.00642194 BTC
Actual Rate:
6,228.63899850 USD
Commission:
0.03999994 USD
Total:
40.03994587 USD

When it tried to close this trade, it sent the following Telegram message:

Bittrex: Selling BTC/USD
Amount: 0.00642194
Open Rate: 6228.63900000
Current Rate: 6242.39700000
Close Rate: 6242.39700000
Sell Reason: sell_signal
Duration: 2:04:54 (124.9 min)
Profit: -0.28% (loss: -0.11186756 USD / -0.112 USD)

When checking Bittrex order history, we can see that it actually made a profit:

Closed Date:
2020/04/01 17:31:43
Opened Date:
2020/04/01 17:15:09
Market:
USD-BTC
Side:
SELL
Type:
Limit
Price:
6,242.39700000 USD
Quantity Filled:
0.00642194 BTC
Total Quantity:
0.00642194 BTC
Actual Rate:
6,242.39699997 USD
Commission:
0.04008829 USD
Total:
40.04821070 USD

If we do 0.00642194 BTC x 6,242.39699997 USD = 40.08829898998734‬ USD If we do 40.08829898998734‬ USD (total money credited to account after sell) - 0.04008829 (sell commission) = 40.04821070 USD - 0.03999994 USD (commission on buy order) = ‭0.00826483‬ USD (our massive profit) which means that we made a profit of 0.02%, not a loss of -0.28%.


I've observed this with other orders, as it can be seen clearly in the order history window, especially when sell order is right after buy order.

In the next screenshot, the bot reported a loss of 0.33%, but the loss was really 1.66%: btcusd1

In the next screenshot, the bot reported a profit of 0.17%, but in fact it was 0.49% btcusd2


Finally, in closing, I want to mention that it happens with other pairs as well, not just USD/BTC.

xmatthias commented 4 years ago

Please check the logs for any message > INFO level which did happen in the timeframe (+-1-2 minutes) of this trade. Maybe some update or whatnot failed.


For the buy side: the Buy telegram message is sent when the buy order is created. Fees are not known at that point - so the buy message is working as intended (it's sending you all information known at that point in time).

for the fee handling itself - this is a known issue to some regard (#2649) - and especially applies when non-standard fees are applied. The issue itself proposes to move this to the users responsibility, which i disagree with - but i didn't yet have time to fully dig into this.


To help in aligning the fee handling between all exchanges, the following data would be helpful - as this should contain the information on fees applied.

import ccxt
from datetime import datetime
ct = ccxt.bittrex({'apiKey': '<key>', 'secret': '<secret>'}
ct.load_markets()
since = int(datetime(2020,4,1).timestamp() * 1000)
res= ct.fetch_my_trades('BTC/USD', since=since)
print(res)

Feel free to remove all trades which are not the above - i'll really just need one buy, and one sell trade (which ideally align).

mostlyvirtual commented 4 years ago

Hello @xmatthias - thank you for your reply.

INFO messages before and after the trade look good to me:

2020-04-01 14:09:20,924 - freqtrade.worker - INFO - Bot heartbeat. PID=3386, version='2020.3', state='RUNNING'
2020-04-01 14:10:06,847 - freqtrade.data.converter - INFO - Missing data fillup for BTC/USD: before: 5759 - after: 5776
2020-04-01 14:10:11,969 - freqtrade.data.converter - INFO - Missing data fillup for BTC/USD: before: 5759 - after: 5776
2020-04-01 14:10:12,161 - freqtrade.wallets - INFO - Wallets synced.
2020-04-01 14:10:12,163 - freqtrade.freqtradebot - INFO - Buy signal found: about create a new trade with stake_amount: 40 ...
2020-04-01 14:10:12,167 - freqtrade.freqtradebot - INFO - Executing Buy for BTC/USD
2020-04-01 14:10:12,167 - freqtrade.freqtradebot - INFO - Using Last Bid / Last Price
2020-04-01 14:10:16,633 - freqtrade.wallets - INFO - Wallets synced.
2020-04-01 14:10:16,639 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'type': buy, 'exchange': 'Bittrex', 'pair': 'BTC/USD', 'limit': 6228.639, 'order_type': 'limit', 'stake_amount': 40.0, 'stake_currency': 'USD', 'fiat_currency': 'USD', 'amount': 0.006421948679318227, 'open_date': datetime.datetime(2020, 4, 1, 12, 10, 15, 150872), 'current_rate': 6228.639}
2020-04-01 14:10:18,358 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=22, pair=BTC/USD, amount=0.00642195, open_rate=6228.63900000, open_since=2020-04-01 12:10:15)
2020-04-01 14:10:23,139 - freqtrade.worker - INFO - Bot heartbeat. PID=3386, version='2020.3', state='RUNNING'
2020-04-01 14:10:23,378 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=22, pair=BTC/USD, amount=0.00642195, open_rate=6228.63900000, open_since=2020-04-01 12:10:15)
2020-04-01 14:10:28,381 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=22, pair=BTC/USD, amount=0.00642195, open_rate=6228.63900000, open_since=2020-04-01 12:10:15)
2020-04-01 14:10:33,368 - freqtrade.freqtradebot - INFO - Found open order for Trade(id=22, pair=BTC/USD

From the sqlite db, this is the trade in question:

22|bittrex|BTC/USD|0|0.0025|0.0025|6228.639|6228.639|40.0999458045092|6242.397|6242.397|-0.00278972|-0.11186756|40.0|0.00642194|2020-04-01 12:10:15.150872|2020-04-01 14:31:49.893829||5917.20705|-0.05|5917.20705|-0.05||2020-04-01 12:13:01.312457|6254.363|6190.406|sell_signal|Strategy005|5

I just now noticed that the 5th and 6th columns are fee_open and fee_closed. Not sure what 0.0025 is, but that is wrong. If it's BTC, then that's $17, so clearly not correct, and I don't suspect it's BTC anyway. If it's USD, which is more likely, it should be 0.1%, which should be roughly 4cents.


Thanks for the explanation about the fee - that makes total sense now that I think of it. The bot cannot know what the fee was until the order has been filled, unless there was a way of setting 0.1% fee in the config.json file, but I guess that's what you meant when you said "to move this to the user's responsibility".


This is the return from the exchange for the ccxt command you asked:

{'id': '518083af-3419-473e-8601-af4cc41cd2fb', 'side': 'buy', 'order': '518083af-3419-473e-8601-af4cc41cd2fb', 'type': 'limit', 'price': 6228.638998495782, 'amount': 0.00642194, 'cost': 39.99994593, 'symbol': 'BTC/USD', 'timestamp': 1585743172950, 'datetime': '2020-04-01T12:12:52.950Z', 'fee': {'cost': 0.03999994, 'currency': 'USD'}, 'info': {'id': '518083af-3419-473e-8601-af4cc41cd2fb', 'timestamp': 1585743015120, 'datetime': '2020-04-01T12:10:15.120Z', 'lastTradeTimestamp': 1585743172950, 'symbol': 'BTC/USD', 'type': 'limit', 'side': 'buy', 'price': 6228.639, 'cost': 39.99994593, 'average': 6228.638998495782, 'amount': 0.00642194, 'filled': 0.00642194, 'remaining': 0.0, 'status': 'closed', 'fee': {'cost': 0.03999994, 'currency': 'USD'}, 'info': {'id': '518083af-3419-473e-8601-af4cc41cd2fb', 'marketSymbol': 'BTC-USD', 'direction': 'BUY', 'type': 'LIMIT', 'quantity': '0.00642194', 'limit': '6228.63900000', 'timeInForce': 'GOOD_TIL_CANCELLED', 'fillQuantity': '0.00642194', 'commission': '0.03999994', 'proceeds': '39.99994593', 'status': 'CLOSED', 'createdAt': '2020-04-01T12:10:15.12Z', 'updatedAt': '2020-04-01T12:12:52.95Z', 'closedAt': '2020-04-01T12:12:52.95Z'}}}, {'id': '7922eeb1-3faa-4024-8b6f-1126501907b3', 'side': 'sell', 'order': '7922eeb1-3faa-4024-8b6f-1126501907b3', 'type': 'limit', 'price': 6242.396999971971, 'amount': 0.00642194, 'cost': 40.08829899, 'symbol': 'BTC/USD', 'timestamp': 1585751503860, 'datetime': '2020-04-01T14:31:43.860Z', 'fee': {'cost': 0.04008829, 'currency': 'USD'}, 'info': {'id': '7922eeb1-3faa-4024-8b6f-1126501907b3', 'timestamp': 1585750509360, 'datetime': '2020-04-01T14:15:09.360Z', 'lastTradeTimestamp': 1585751503860, 'symbol': 'BTC/USD', 'type': 'limit', 'side': 'sell', 'price': 6242.397, 'cost': 40.08829899, 'average': 6242.396999971971, 'amount': 0.00642194, 'filled': 0.00642194, 'remaining': 0.0, 'status': 'closed', 'fee': {'cost': 0.04008829, 'currency': 'USD'}, 'info': {'id': '7922eeb1-3faa-4024-8b6f-1126501907b3', 'marketSymbol': 'BTC-USD', 'direction': 'SELL', 'type': 'LIMIT', 'quantity': '0.00642194', 'limit': '6242.39700000', 'timeInForce': 'GOOD_TIL_CANCELLED', 'fillQuantity': '0.00642194', 'commission': '0.04008829', 'proceeds': '40.08829899', 'status': 'CLOSED', 'createdAt': '2020-04-01T14:15:09.36Z', 'updatedAt': '2020-04-01T14:31:43.86Z', 'closedAt': '2020-04-01T14:31:43.86Z'}}}

This is the same trade as outlined in my original example, and the one that I extracted from the sqlite db - I used the same one to consistency.

I used a json to csv convertor to make this easier to read for me - not sure if it helps you, but I attached the file anyway. result.xlsx

xmatthias commented 4 years ago

fee_open and fee_closed are as ratio - so 0.0025 would be 0.25% ... (now the problem is - taken from which side ... and from which ... - therefore i'm currently gathering all possible returns from exchanges).

Json data is perfectly fine ... that's actually easier to automatically read than csv / xlsx - and i don't really need it "readable" - that's what the screenshots provide - i need to know how the actual result from the API looks like.

IpsissimusMarr commented 4 years ago

For Bittrex at least there is a discrepancy between the fees listed for markets and actual fees recorded. For instance, running ct.fetch_my_trades('BTC/USD' returns the following:

 {"BTC/USD":{"percentage":true,"tierBased":false,"maker":0.0025,"taker":0.0025,"precision":{"amount":8,"price":3},"limits":{"amount":{"min":0.00025427,"max":"None"},"price":{"min":0.001,"max":"None"}},"id":"USD-BTC","symbol":"BTC/USD","base":"BTC","quote":"USD","baseId":"BTC","quoteId":"USD","active":true,"info":{"symbol":"BTC-USD","baseCurrencySymbol":"BTC","quoteCurrencySymbol":"USD","minTradeSize":"0.00025427","precision":3,"status":"ONLINE","createdAt":"2018-05-31T13:24:40.77Z","prohibitedIn":[]}}}

Which lists fees as 0.0025. Of course this can change based on volume or discounts, but with a recent trade I calculate a fee of 0.001 based on:

{"id":"XXX","side":"buy","order":"XXX","type":"limit","price":7083.515999929165,"amount":0.00282345,"cost":19.99995325,"symbol":"BTC/USD","timestamp":1587565352960,"datetime":"2020-04-22T14:22:32.960Z","fee":{"cost":0.01999995,"currency":"USD"},"info":{"id":"XXX","timestamp":1587565310790,"datetime":"2020-04-22T14:21:50.790Z","lastTradeTimestamp":1587565352960,"symbol":"BTC/USD","type":"limit","side":"buy","price":7083.516,"cost":19.99995325,"average":7083.515999929165,"amount":0.00282345,"filled":0.00282345,"remaining":0,"status":"closed","fee":{"cost":0.01999995,"currency":"USD"},"info":{"id":"XXX","marketSymbol":"BTC-USD","direction":"BUY","type":"LIMIT","quantity":"0.00282345","limit":"7083.51600000","timeInForce":"GOOD_TIL_CANCELLED","fillQuantity":"0.00282345","commission":"0.01999995","proceeds":"19.99995325","status":"CLOSED","createdAt":"2020-04-22T14:21:50.79Z","updatedAt":"2020-04-22T14:22:32.96Z","closedAt":"2020-04-22T14:22:32.96Z"}}}

Which is a purchase of 19.99995325 with 0.01999995 fee for a calculated 0.001.

The same is true for the sell side for the corresponding buy:

{"id":"XXX","side":"sell","order":"XXX","type":"limit","price":7120.421997910358,"amount":0.00282345,"cost":20.10415549,"symbol":"BTC/USD","timestamp":1587566339830,"datetime":"2020-04-22T14:38:59.830Z","fee":{"cost":0.02010415,"currency":"USD"},"info":{"id":"XXX","timestamp":1587566339830,"datetime":"2020-04-22T14:38:59.830Z","lastTradeTimestamp":1587566339830,"symbol":"BTC/USD","type":"limit","side":"sell","price":7120.42,"cost":20.10415549,"average":7120.421997910358,"amount":0.00282345,"filled":0.00282345,"remaining":0,"status":"closed","fee":{"cost":0.02010415,"currency":"USD"},"info":{"id":"XXX","marketSymbol":"BTC-USD","direction":"SELL","type":"LIMIT","quantity":"0.00282345","limit":"7120.42000000","timeInForce":"GOOD_TIL_CANCELLED","fillQuantity":"0.00282345","commission":"0.02010415","proceeds":"20.10415549","status":"CLOSED","createdAt":"2020-04-22T14:38:59.83Z","updatedAt":"2020-04-22T14:38:59.83Z","closedAt":"2020-04-22T14:38:59.83Z"}}}

Which is 20.10415549 sell for 0.02010415 fee to get 0.001 again. This later in inteface.stop_loss_reached() was calculated as a profit of -0.03642967. When freqtrade gets the order back from the exchange wouldn't it be better to extract and use the actual fees recorded so profit and stoploss calculations can be accurate?

xmatthias commented 4 years ago

Let me repeat what i said above: this is a known issue for the most part (#2649) - and i hope to have some time this weekend to finish the implementation for proper fee handling.

mostlyvirtual commented 4 years ago

@xmatthias do you need me to test this? is it in the default develop branch?

xmatthias commented 4 years ago

no it's not, it's in the feat/fee_handling branch.

I've been testing it myself and it seems ok so far - but my time for that is currently limited, so help in testing it is highly apreciated.

mostlyvirtual commented 4 years ago

@xmatthias I started a bot now on that branch: 2020-05-08 22:01:20,135 - freqtrade.worker - INFO - Bot heartbeat. PID=4546, version='develop-e92d3867', state='RUNNING'

I'll let you know after it does a few trades, but it could be a few days.

Thanks.

hroff-1902 commented 4 years ago

the fix was merged into develop, pls verify

mostlyvirtual commented 4 years ago

I only had one order since moving to develop-e92d3867 and it was closed correctly - i.e. all the amounts matched perfectly with my Bittrex panel.

67|bittrex|HBAR/USD|0|0.002|0.08000002|USD|0.002|0.08031528|USD|0.03535069|0.03535068|40.080011337606|0.03549|0.03549|-6.695e-05|-0.00268326|40.0|1131.51995944|2020-05-12 23:30:10.737901|2020-05-13 15:34:51.345353||0.0335831555|-0.05|0.0335831555|-0.05||2020-05-12 23:33:29.099709|0.03636306|0.03499998|sell_signal|Strategy005|5

I'll update this bot to develop branch and restart it. If you want, you can keep this issue closed for now, but I'll keep an eye out for future trades, until this makes it into master.

hroff-1902 commented 4 years ago

sure, thanks 👍