eprbell / rp2

Privacy-focused, free, open-source cryptocurrency tax calculator for multiple countries: it handles multiple coins/exchanges and computes long/short-term capital gains, cost bases, in/out lot relationships/fractioning, and account balances. It supports FIFO, LIFO, HIFO and it outputs in form 8949 format. It has a programmable plugin architecture
https://pypi.org/project/rp2/
Apache License 2.0
278 stars 44 forks source link

Crypto fee in OUT table #82

Closed gbtorrance closed 1 year ago

gbtorrance commented 1 year ago

The OUT Table format indicates that "crypto_fee" is required.

It seems that almost all of my transactions had SELL fees paid in fiat. I can calculate a "crypto_fee" based on "fiat_fee" and "spot_price", but I wonder how RP2 handles this.

Is RP2 assuming the fee was paid in crypto? And, if so, is it reducing the crypto balance and cost basis remaining by the amount of "crypto_fee" (in addition to the amount of "crypto_out_no_fee")? In my case that would NOT be correct, right?

Can you help me understand? Thanks!

eprbell commented 1 year ago

Take a look at this FAQ, which covers fee modeling: https://github.com/eprbell/rp2/blob/main/docs/user_faq.md#how-to-represent-fiat-vs-crypto-transaction-fees

gbtorrance commented 1 year ago

OK, I think I understand. So in my case, I should just set "crypto_fee" to 0 and record the actual fee in "fiat_fee"? Is there a reason that "crypto_fee" is required for OUT but optional for IN? (That's part of why I was confused. Figured I had to enter a non-zero crypto fee value.)

Thanks for the info!

eprbell commented 1 year ago

No worries! The optional vs non-optional fee issue was discussed here: https://github.com/eprbell/dali-rp2/pull/75. I'm open to reviewing PRs if somebody wants to make it optional everywhere.

gbtorrance commented 1 year ago

OK, thanks again. (Closing.)

gbtorrance commented 1 year ago

Hello again @eprbell. Hope you don't mind me re-opening this issue.

After about a month of being distracted on other tasks, I'm back to using RP2. I've encountered a problem related to crypto_fee. My OUT transactions have the fee represented in FIAT, so I understand I should set crypto_fee (which is required) to 0, and set fiat_fee to the actual fee in USD. I have done this, but now I'm getting the following warnings:

INFO: Processing ADA
WARNING: ADA OutTransaction (2022-01-12 19:57:59.915000+00:00, id 160): crypto_out_no_fee * spot_price != fiat_out_no_fee: 10.343055 != 7.990000
WARNING: ADA OutTransaction (2022-01-13 00:33:12.844000+00:00, id 161): crypto_out_no_fee * spot_price != fiat_out_no_fee: 10.343513 != 7.730000
WARNING: ADA OutTransaction (2022-01-16 10:36:30.491000+00:00, id 162): crypto_out_no_fee * spot_price != fiat_out_no_fee: 181.048770 != 129.450000

Though I understand these are just warnings and not errors, they are a bit concerning, and seem to contradict the fee approach described here. The only way I can get rid of these warnings is to calculate crypto_fee so it is fiat_fee / spot_price, but then I assume that RP2 is reducing my crypto holdings by the amount of crypto_fee, which is not the desired behavior.

Thoughts? Thanks!

eprbell commented 1 year ago

Can you attach the 3 transactions as they appear in the spreadsheet? Feel free to anonymyze them, if you wish.

gbtorrance commented 1 year ago

Oops, seems I posted the wrong log information above. What I accidentally posted was from an earlier run where I had some incorrect formulas in my spreadsheet. What I meant to post is this:

WARNING: ADA OutTransaction (2022-01-12 19:57:59.915000+00:00, id 160): crypto_fee * spot_price != fiat_fee: 0.000000 != 0.036201
WARNING: ADA OutTransaction (2022-01-13 00:33:12.844000+00:00, id 161): crypto_fee * spot_price != fiat_fee: 0.000000 != 0.036202
WARNING: ADA OutTransaction (2022-01-16 10:36:30.491000+00:00, id 162): crypto_fee * spot_price != fiat_fee: 0.000000 != 0.633671

And here are the actual OUT transactions: image

Hope that makes sense. Thanks.

eprbell commented 1 year ago

Thanks for the data. Just to be sure we're on the same page: this is the case of an exchange that charges fiat fees when selling crypto, correct? I haven't seen this before, so this is interesting.

gbtorrance commented 1 year ago

Yes, I believe that is the case, but I went back to confirm. I am actually using a normalized dataset I extracted from CoinLedger.io, which allows me to rely on CoinLedger's import facility to gather all my transactions. In that dataset it indicates that Coinbase Pro transaction fees are in USD. But I logged into Coinbase Pro to confirm, and yes indeed, that seems to be the case:

image

FWIW the transactions in question where USD pairs rather than stablecoin pairs, so that could be part of the reason? (Don't know if it operates differently for stablecoin paris.) And I also don't know if Coinbase (NOT Pro) operates differently. As I understand it, Coinbase Pro may be going way and being absorbed fully into regular Coinbase.

eprbell commented 1 year ago

In Coinbase Pro the fees are paid in crypto, not in fiat. I know: it's confusing because the Coinbase Pro CSV shows a USD value, but if you check the crypto balance before and after the sale, you'll see that the amount that came out corresponds to Qty + (Fee / Price).

You can also check the DaLI Coinbase Pro data loader source code, which converts the fiat fee to crypto: https://github.com/eprbell/dali-rp2/blob/main/src/dali/plugin/input/rest/coinbase_pro.py#L347

So the warning is correct: it is there to alert the user to a potential problem with the data.

gbtorrance commented 1 year ago

Thanks for the info. OK, so that's very good to know. Thanks, Coinbase Pro, for making life more complicated than it needs to be :-(

So the warning is correct: it is there to alert the user to a potential problem with the data.

While this may be true in this particular case, just to play devil's advocate, one could argue that the warning does actually make it impossible to support the first bullet point here without receiving the warning. Do you consider the first bullet point a legitimate use case? Anyway, just something to consider :-)

Thanks again for all the help! Please feel free to close the ticket if you're ready to do so.

eprbell commented 1 year ago

Devil's advocates are welcome here! :-) I am not aware of exchanges that charge fiat fees for out-transactions and that's what the warning tries to capture. If such a case appears, we can adjust the first bullet and the code accordingly, but until then I'd rather issue a warning conservatively, to alert the user to a potential problem with their data.

eprbell commented 1 year ago

Closing for now.

gbtorrance commented 1 year ago

@eprbell Makes total sense!

gbtorrance commented 1 year ago

Hi @eprbell. Me again. Sorry to be a pain.

In Coinbase Pro the fees are paid in crypto, not in fiat. I know: it's confusing because the Coinbase Pro CSV shows a USD value, but if you check the crypto balance before and after the sale, you'll see that the amount that came out corresponds to Qty + (Fee / Price).

You can also check the DaLI Coinbase Pro data loader source code, which converts the fiat fee to crypto: https://github.com/eprbell/dali-rp2/blob/main/src/dali/plugin/input/rest/coinbase_pro.py#L347

So the warning is correct: it is there to alert the user to a potential problem with the data.

I am once again working on my taxes, and once again I keep coming back to this particular issue.

No matter what I do -- and I've tried a number of different permutations of my spreadsheet -- I keep arriving at the conclusion that Coinbase Pro is indeed taking the fee in USD and not in crypto.

Part of the reason I'm convinced of this is that I closed out many different "alt coin" positions last year (to zero or "dust" remaining), and whenever I record the fee as a crypto fee, my balance runs out when running RP2 and I get the "Total in-transaction crypto value < total taxable crypto value" error. When I calculate the sum total of crypto_in and crypto_out_no_fee for that position, they are basically identical, with crypto_fee being enough to cause the error. But when I adjust my spreadsheet to treat the fee as a USD fee, and set crypt_fee to 0, it seems to work, and I no longer get the error.

But I do get a flood of warnings, such as these:

WARNING: ATOM OutTransaction (2022-01-02 22:37:45.498000+00:00, id 134): crypto_fee * spot_price != fiat_fee: 0.000000 != 0.159210
WARNING: ATOM OutTransaction (2022-01-03 08:05:25.903000+00:00, id 135): crypto_fee * spot_price != fiat_fee: 0.000000 != 0.138801
WARNING: ATOM OutTransaction (2022-01-03 22:11:27.458000+00:00, id 136): crypto_fee * spot_price != fiat_fee: 0.000000 != 0.023196

Some more "proof", FWIW: I tried a test this morning in my Coinbase Pro account. I only have about $2 USD in that account, so I decided to buy and sell some ETH. First I bought 0.00122558 in ETH and used all of my remaining USD. The fee was $0.01. I checked my portfolio and confirmed USD = 0 and ETH = 0.00122558. Then I sold the full ETH balance of 0.00122558. The fee was $0.01. I checked my portfolio and confirmed that ETH = 0 and USD = $1.96. The USD balance isn't all that important, but the thing that seems relevant to me is that I was able to sell my entire ETH balance, which shouldn't be possible if the fee were being taken in ETH rather than USD. Seems to me the fee is being taken from the USD proceeds of the sale, not the crypto balance.

image

To summarize, I think it's worth reconsidering whether the above warnings are appropriate, and whether DALI is calculating the fee correctly for Coinbase Pro.

Thoughts?

Thanks again for RP2 and everything you do on the project. It's a life saver!

eprbell commented 1 year ago

The example you gave shows the following:

As for the "Total in-transaction crypto value < total taxable crypto value": this usually denotes an error in the data that was entered. I suggest you to try generating your Coinbase Pro RP2 input with DaLI (https://github.com/eprbell/dali-rp2) and see what the difference is.

Another good practice is to check the "Account Balances" in the ETH Tax tab of the rp2_full_report.ods output: the balance should match what you see in your account. If it doesn't it's probably due to errors in the input data.

gbtorrance commented 1 year ago

@eprbell Thanks for the reply.

... must have come out of your crypto proceeds (again, because you had no USD).

Yes, absolutely! Totally agree! (But the word "proceeds" is critical to that statement.)

The amount of the sale fee can be displayed in ETH or in USD (Coinbase Pro shows it in USD), but it is coming out of the ETH you sold, therefore it's a crypto fee.

Yes, the fee can be displayed in ETH or USD. But the fee is coming from the proceeds of the sale (as you noted above). How does it follow that "therefore it's a crypto fee"?

Whether the fee is taken from the USD balance in the account at the time of sale (in this particular case, my balance was $0) or from the proceeds of the sale (which will immediately be added to the USD balance in the account the moment the order fills), it's still a USD fee. Maybe it's an issue of interpretation, but IMO all "reasonable evidence" indicates this should be treated as a fiat fee:

IMO, there is nothing here to make it logical to interpret this as a crypto fee rather than a fiat fee. It would require "bending over backwards" to arrive at numbers to make this work, when it's trivially easy to just interpret it at face value, which is that we're dealing with a fiat fee. (To interpret as a crypto fee I'd need to both convert the $0.01 USD fee to crypto and then subtract that from the 0.00122558 ETH sale quantity reported by Coinbase Pro to arrive at crypto_out_no_fee, since the sum of crypto_out_no_fee and crypto_fee must add up to 0.00122558 ETH. Crazy complicated when it really doesn't need to be! Rather just treat 0.00122558 ETH as crypto_out_no_fee and $0.01 as fiat_fee and it just works. The only complication is the above warning and the fact that crypto_fee is a required field that needs to be set to 0.)

I'm really not trying to be difficult, but I do think that viewing the fee (not just for Coinbase, but probably all exchanges) as coming from the fiat proceeds of the sale rather than from the crypto itself is a much more intuitive interpretation of what's actually happening.

Thoughts?

eprbell commented 1 year ago

The semantics of crypto fee and fiat fee in RP2 are as follows:

The case you described falls in the first category. Again, I encourage you to:

gbtorrance commented 1 year ago

Thanks @eprbell.

I tried DaLI, but got the following exception:

$ dali_us test_config.ini
INFO: Country: us
INFO: Initialized input plugin 'dali.plugin.input.rest.coinbase_pro'
INFO: No pair converter plugins found in configuration file: using default pair converters.
INFO: Reading crypto data using plugin 'dali.plugin.input.rest.coinbase_pro'
ERROR: Fatal exception occurred:
Traceback (most recent call last):
  File "/home/mint21/.local/lib/python3.10/site-packages/dali/dali_main.py", line 163, in _dali_main_internal
    result_list = pool.map(_input_plugin_helper, input_plugin_args_list)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/mint21/.local/lib/python3.10/site-packages/dali/dali_main.py", line 199, in _input_plugin_helper
    plugin_transactions = input_plugin.load()
  File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 157, in load
    process_account_result_list = pool.map(self._process_account, accounts)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 185, in _process_account
    self._process_transfer(transaction, currency, intra_transaction_list)
  File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 224, in _process_transfer
    transfer_id: str = transaction_details[_TRANSFER_ID]
KeyError: 'transfer_id'
INFO: Log file: ./log/rp2_2023_01_29_14_07_29_876609.log
INFO: Generated output directory: output/
INFO: Done

check the "Account Balances" as I described above.

I have looked at some of the account balances for some of my positions, and I think they are at least in the ballpark. Unfortunately, I have had almost 50k transactions (due to bot trading) across 5 - 10 exchange accounts, so it will be far from easy to know if the balances are 100% correct. If I can feel comfortable that they are "close to correct" I'll be happy. (That said, I want to do whatever I can to validate correctness on a smaller data set.)

The semantics of crypto fee and fiat fee in RP2 are as follows:

  • crypto fee means that a part of the crypto you just sold in a transaction is used to pay the fee,
  • fiat fee means that the entirety of the crypto gets sold and the fee comes out of your fiat stash.

We can probably go back and forth on this indefinitely, but I still feel that the second option is the closer to being correct. A crypto sale will always result in a non-zero fiat balance, but may result in a zero crypto balance. IMO, it makes more sense to think of the fee coming from the guaranteed non-zero fiat balance than from the possibly zero crypto balance (especially if the exchange explicitly tells you it's a fiat fee). Yes, I understand that you view the fee as coming from the actual crypto that is being sold (which is guaranteed to be non zero), but I still feel it makes more sense to simply view it as selling the entirety of the crypto and paying the fiat fee from the proceeds of the sale. Coinbase Pro says this is what happened. What reason is there to make this into something more complicated and confusing?

BTW, I checked another one of my exchanges (KuCoin), and it also reports a fiat fee for sells:

image

eprbell commented 1 year ago

The second option is not "closer to being correct": the use case you described doesn't match the semantics of the second option and it matches those of the first option. I gave the definition of what crypto fee means in RP2 (its operational semantics) and what you described falls into the definition of crypto fee in RP2. You may not like the definition but RP2 has been designed with that definition from the beginning: if part of the crypto sold is used to pay the fee, then that partial amount is called crypto fee in RP2. Again, this is how RP2 defines that concept (it's not right or wrong, it's just a definition). How exchanges choose to represent that fee is arbitrary, meaning that different exchanges can (and do) make different choices. However it's essential for tax software to choose a definition and stick to it. This is why DaLI is useful: it converts the arbitrary representations of exchanges into the common/normalized format of RP2.

As for the DaLI stack trace you pasted: it looks like some implementation is missing in the context of a Coinbase Pro transfer. Would you be able to:

If you can paste here that line I'll take a look (feel free to anonymize personally identifiable details).

gbtorrance commented 1 year ago

I'm really not as dense as you probably think I am :-)

if part of the crypto sold is used to pay the fee, then that partial amount is called crypto fee in RP2.

The way I see it, this is what happened:

Why am I so convinced that the fee is a fiat fee and not a crypto fee?

However it's essential for tax software to choose a definition and stick to it.

I can appreciate that. However you also listed the following as an option:

fiat fee means that the entirety of the crypto gets sold and the fee comes out of your fiat stash.

If what I'm describing is not this, then when could this ever occur?

I maintain this is exactly what is happening -- probably for the majority of sell transactions on most exchanges -- with the understanding that one's "fiat stash" includes the proceeds of the sell transaction.

To be clear, I don't think there is any significant issue with RP2. The only potential concern, if there is one, is that the crypto_fee * spot_price != fiat_fee warning cannot be prevented when crypto_fee = 0 and fiat_fee > 0. It really comes down to a different perspective on what is happening when a sell occurs. RP2 supports both. It's just that I feel that the evidence supports "fiat fee", whereas you seem to feel it's "crypto fee". I am nobody. It's your project. It's just that I have spent days trying to "pretzel" my transaction history into a format that produces accurate results using "crypto fee", and I haven't been able to do so yet. But as "fiat fee" ... it just works. That's how it's reported by Coinbase, and I feel that's actually what's happening behind the scenes. It makes a lot of sense to me.

As requested, here is the log info. Hope it helps.

2023-01-29 15:28:09,665/Coinbase Pro/Greg/DEBUG: Transfer is a Coinbase transaction already captured by Coinbase plugin: ignoring.
2023-01-29 15:28:09,666/Coinbase Pro/Greg/DEBUG: Transaction: {"id": "11275304484", "amount": "12.8210000000000000", "balance": "12.8210000000000000", "created_at": "2021-10-09T15:11:55.187005Z", "type": "transfer", "details": {"from": "ccbef35f-d0c3-4a80-8ec5-f7e701f58ea1", "profile_transfer_id": "19d52363-3563-4398-8e6f-66c039b1bcff", "to": "6c7f97c7-4deb-497f-9f02-99d5b91038db"}}
2023-01-29 15:28:09,670/dali/ERROR: Fatal exception occurred:
Traceback (most recent call last):
...
KeyError: 'transfer_id'

Thanks again.

eprbell commented 1 year ago

The way I see it, this is what happened:

* Me: Coinbase, please sell my 0.00122558 ETH.

* CB: OK, done. We sold your 0.00122558 ETH. At a spot price of $1613.01 that leaves you with sale proceeds of $1.97. But we need to take our cut of $0.01, so we added $1.96 to the USD balance in your account. Pleasure doing business with you.

* Me: Thank you.

Why am I so convinced that the fee is a fiat fee and not a crypto fee?

* Coinbase (and KuCoin) **say** it's a fiat fee.

* Exchanges have absolutely no reason to want thousands of alt coin "fees" on their books for accounting purposes. They want fiat on their books. So if they were to take part of the crypto as a fee, they would almost certainly immediately sell it themselves for fiat. But I can't see why they would do that if they could simply take a fiat fee from the proceeds of the sale (as vividly illustrated above ;-) )

That may be, but, again, if you look at my definition, crypto fee in RP2 means: you sell your 0.00122558 ETH and part of that is used to pay the fee. Whether the exchange internally converts the crypto fee to fiat is immaterial to the definition: the fee comes out of the crypto amount in the first place.

However it's essential for tax software to choose a definition and stick to it.

I can appreciate that. However you also listed the following as an option:

fiat fee means that the entirety of the crypto gets sold and the fee comes out of your fiat stash.

If what I'm describing is not this, then when could this ever occur?

If you sold your 0.00122558 ETH and then the exchange got the fee from your preexisting fiat stash (in your case it is 0 USD, so this couldn't happen) without subtracting it from your 0.00122558 ETH, that would be an example of RP2 fiat fee.

I maintain this is exactly what is happening -- probably for the majority of sell transactions on most exchanges -- with the understanding that one's "fiat stash" includes the proceeds of the sell transaction.

To be clear, I don't think there is any significant issue with RP2. The only potential concern, if there is one, is that the crypto_fee * spot_price != fiat_fee warning cannot be prevented when crypto_fee = 0 and fiat_fee > 0. It really comes down to a different perspective on what is happening when a sell occurs. RP2 supports both. It's just that I feel that the evidence supports "fiat fee", whereas you seem to feel it's "crypto fee". I am nobody. It's your project. It's just that I have spent days trying to "pretzel" my transaction history into a format that produces accurate results using "crypto fee", and I haven't been able to do so yet. But as "fiat fee" ... it just works. That's how it's reported by Coinbase, and I feel that's actually what's happening behind the scenes. It makes a lot of sense to me.

I appreciate the feedback and I heard you but I disagree with your premise and it sounds like we're beginning to go in circles. Crypto fee in this context is an RP2 concept which is defined as I explained above (again, it's just a definition).

As requested, here is the log info. Hope it helps.

2023-01-29 15:28:09,665/Coinbase Pro/Greg/DEBUG: Transfer is a Coinbase transaction already captured by Coinbase plugin: ignoring.
2023-01-29 15:28:09,666/Coinbase Pro/Greg/DEBUG: Transaction: {"id": "11275304484", "amount": "12.8210000000000000", "balance": "12.8210000000000000", "created_at": "2021-10-09T15:11:55.187005Z", "type": "transfer", "details": {"from": "ccbef35f-d0c3-4a80-8ec5-f7e701f58ea1", "profile_transfer_id": "19d52363-3563-4398-8e6f-66c039b1bcff", "to": "6c7f97c7-4deb-497f-9f02-99d5b91038db"}}
2023-01-29 15:28:09,670/dali/ERROR: Fatal exception occurred:
Traceback (most recent call last):
...
KeyError: 'transfer_id'

Thanks again.

Thanks for this data: it is indeed a transfer that doesn't have "transfer_id" as a key, which I haven't seen before. Could you provide some clarification on this transaction? What were you trying to do: perhaps a transfer to Coinbase? Thanks.

eprbell commented 1 year ago

I just realized we already ran into and discussed this issue: based on the stack trace and the logs, it looks like another instance of DaLI's issue 82. Is that a transfer between CBPro portfolios? If so, that is a missing bit of implementation in the CBPro plugin (no portfolio support), but I can't implement it because I don't have CBPro portfolios, so it would have to be contributed by somebody who does. In any case, thanks for reporting it.

gbtorrance commented 1 year ago

Hi @eprbell. Thanks for the replies.

I just realized we already ran into and discussed this issue

Yes, you're absolutely right about the portfolios. That's almost certainly the case.

I appreciate the feedback and I heard you but I disagree with your premise and it sounds like we're beginning to go in circles.

I get that. I'm sorry to keep pushing the issue. If I may just make one more "final statement" then I'll get out of your hair: IMO, fiat fee on sell transactions always comes from the fiat proceeds of the sale. Any pre-existing crypto or fiat balances in the account are entirely irrelevant. Expecting this to be reported in RP2 using complex translations like the following make something which is actually quite simple (crypto_out_no_fee and fiat_fee as reported by the exchange in Qty and Fee) into something unnecessarily complicated and confusing (as evidenced by the length of this thread):

image

I'll leave it at that. Thank you for hearing me out. I appreciate it :-)