search

eprbell / dali-rp2

DaLI (Data Loader Interface) is a data loader and input generator for RP2 (https://pypi.org/project/rp2), the privacy-focused, free, open-source cryptocurrency tax calculator: DaLI removes the need to manually prepare RP2 input files. Just like RP2, DaLI is also free, open-source and it prioritizes user privacy.
https://pypi.org/project/dali-rp2/
Apache License 2.0
64 stars 42 forks source link

Algorand Interest Generated with $0 Spot Price #19

Closed jameskupke closed 2 years ago

jameskupke commented 2 years ago

Dali Version: 0.3.25 RP2 Version: 0.9.25

I'm not sure if this is this is an issue with DaLI or with RP2, But Algorand interest is listed in the IN section like the following:

Timestamp Asset Exchange Holder Transaction Type   Spot Price Crypto In Crypto Fee USD In No Fee USD In With Fee USD Fee Unique ID Notes
2021-XX-XX 12:22:48 +0000 ALGO Coinbase ((name)) Income $0.00 0.00012800 $0.00 $0.00 $0.00 ((removed)) Algorand reward
2021-XX-XX 12:15:39 +0000 ALGO Coinbase ((name)) Income $0.00 0.00024200 $0.00 $0.00 $0.00 ((removed)) Algorand reward
2021-XX-XX 14:07:58 +0000 ALGO Coinbase ((name)) Income $0.00 0.00024200 $0.00 $0.00 $0.00 ((removed)) Algorand reward

When trying to run with FIFO or LIFO, I get the following error:

2022-03-27 19:56:10,271/rp2/INFO: Country: us
2022-03-27 19:56:10,273/rp2/INFO: Accounting Method: lifo
2022-03-27 19:56:10,290/rp2/INFO: Configuration file: output/crypto_data.config
2022-03-27 19:56:10,290/rp2/INFO: Input file: output/crypto_data.ods
2022-03-27 19:56:10,361/rp2/INFO: Processing ALGO
2022-03-27 19:56:10,389/rp2/ERROR: Fatal exception occurred:
Traceback (most recent call last):
  File "/home/james/.local/lib/python3.10/site-packages/rp2/rp2_main.py", line 93, in _rp2_main_internal
    input_data: InputData = parse_ods(configuration=configuration, asset=asset, input_file_handle=input_file_handle)
  File "/home/james/.local/lib/python3.10/site-packages/rp2/ods_parser.py", line 128, in parse_ods
    _create_and_process_transaction(
  File "/home/james/.local/lib/python3.10/site-packages/rp2/ods_parser.py", line 168, in _create_and_process_transaction
    transaction: AbstractTransaction = _create_transaction(configuration, current_table_type, internal_id, row_values)
  File "/home/james/.local/lib/python3.10/site-packages/rp2/ods_parser.py", line 277, in _create_transaction
    transaction = InTransaction(**argument_pack)
  File "/home/james/.local/lib/python3.10/site-packages/rp2/in_transaction.py", line 61, in __init__
    raise RP2ValueError(f"{self.asset} {type(self).__name__} ({self.timestamp}, id {self.internal_id}): parameter 'spot_price' cannot be 0")
rp2.rp2_error.RP2ValueError: ALGO InTransaction (2021-04-28 12:22:48+00:00, id 4): parameter 'spot_price' cannot be 0
2022-03-27 19:56:10,400/rp2/INFO: Log file: ./log/rp2_2022_03_27_19_56_10_254635.log
2022-03-27 19:56:10,401/rp2/INFO: Generated output directory: output/
2022-03-27 19:56:10,401/rp2/INFO: Done
eprbell commented 2 years ago

I'm guessing it's probably because Coinbase repports a native_amount == 0: could you paste the DaLI debug log line corresponding to that transaction? You can find it as follows:

Then paste here the line and I'll take a look.

eprbell commented 2 years ago

I guess, you already pasted a similar transaction in issue #16:

2022-03-26 08:27:11,229/Coinbase/name/DEBUG: Transaction: {"id": "ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ", "type": "inflation_reward", "status": "completed", "amount": {"amount": "0.000166", "currency": "ALGO"}, "native_amount": {"amount": "0.00", "currency": "USD"}, "description": null, "created_at": "2021-XX-XXT14:55:27Z", "updated_at": "2021-XX-XXT14:55:27Z", "resource": "transaction", "resource_path": "/v2/accounts/135c87f3-44a8-5c13-ae9c-14ac0d182a79/transactions/ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ", "instant_exchange": false, "from": {"id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", "resource": "user", "resource_path": "/v2/users    /XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX", "currency": "ALGO"}, "details": {"title": "Algorand reward", "subtitle": "From Coinbase", "header": "Received 0.000166 ALGO ($0.00)", "health": "positive"}, "hide_native_amount": false}

DaLI computes the spot_price here: if native_amount/amount is 0 (which it is here, I imagine due to a rounding error), then spot_price will end up being zero. It doesn't look like there is any other data that Coinbase is giving us to figure the correct spot_price. I have an idea about how to resolve this: I'll try it out tomorrow and report back here.

eprbell commented 2 years ago

@jameskupke, I just uploaded a new version that should fix the problem: however you need to pass the -s CLI option. This option reads spot_price information from the Web if it's not available (or if it's 0, like in this case). The switch was available in previous versions, too, but it was using Yahoo Finance, which had 1-day time granularity. The new verison uses the Coinbase Pro API which has 1 min granularity. Let me know if this solves the problem, thanks.

jameskupke commented 2 years ago

Hi @eprbell:

I just got around to pasting a transaction. Looks like the spot price may be really small? "0E+4"

2022-03-27 18:48:53,963/dali/DEBUG: Self-contained transaction: InTransaction:
  plugin=Coinbase
  unique_id=AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA
  raw_data={"id": "AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA", "type": "inflation_reward", "status": "completed", "amount": {"amount": "0.000166", "currency": "ALGO"}, "native_amount": {"amount": "0.00", "currency": "USD"}, "description": null, "created_at": "2021-XX-XXT15:40:22Z", "updated_at": "2021-XX-XXT15:40:22Z", "resource": "transaction", "resource_path": "/v2/accounts/BBBBBBBB-BBBB-BBBB-BBBB-BBBBBBBBBBBB/transactions/AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA", "instant_exchange": false, "from": {"id": "CCCCCCCC-CCCC-CCCC-CCCC-CCCCCCCCCCCC", "resource": "user", "resource_path": "/v2/users/CCCCCCCC-CCCC-CCCC-CCCC-CCCCCCCCCCCC", "currency": "ALGO"}, "details": {"title": "Algorand reward", "subtitle": "From Coinbase", "header": "Received 0.000166 ALGO ($0.00)", "health": "positive"}, "hide_native_amount": false}
  timestamp=2021-XX-XX 15:40:22 +0000
  asset=ALGO
  exchange=Coinbase
  holder=James Kupke
  transaction_type=Income
  spot_price=0E+4
  crypto_in=0.000166
  fiat_fee=0
  fiat_in_no_fee=0.00
  fiat_in_with_fee=0.00
  notes=Algorand reward

For reference here's the Coinbase raw CSV file for some of the transactions: (All the transaction IDs are removed, but they're normally in the first column)

Transaction ID,Transaction Type,Date & time,Asset Acquired,"Quantity Acquired (Bought, Received, etc)",Cost Basis (incl. fees paid) (USD),Data Source,"Asset Disposed (Sold, Sent, etc)",Quantity Disposed,Proceeds (excl. fees paid) (USD)
,Interest,2021-04-29T12:15:39Z,ALGO,0.000242,0.0003323386,Coinbase,,,
,Interest,2021-04-30T14:07:58Z,ALGO,0.000242,0.0003394897,Coinbase,,,
,Interest,2021-05-01T12:24:16Z,ALGO,0.000242,0.000336985,Coinbase,,,
,Interest,2021-05-02T07:42:50Z,DAI,0.00005544,0.000055545336,Coinbase,,,
,Interest,2021-05-02T10:59:38Z,ALGO,0.000242,0.0003308261,Coinbase,,,
eprbell commented 2 years ago

Can you verify for me the following:

The spot prices of last year should be between $0.4 and $3, based on: https://www.coinbase.com/price/algorand

Can you check the following in the generated ODS file:

The new spot_price-reading code is working ok for me, but I don't have Algorand transactions. Strange...

jameskupke commented 2 years ago

Oh no, still using 0.3.25. Will run again with newer version.

eprbell commented 2 years ago

No worries, the new version should output the correct value (but remember to use the -sswitch.

jameskupke commented 2 years ago

Reran with the latest version with the -s switch.

Spot prices are now non zero values, but "USD In No Fee" and "USD In No Fee" are $0.00

This causes this error when running RP2:

INFO: Country: us
INFO: Accounting Method: fifo
INFO: Configuration file: output/crypto_data.config
INFO: Input file: output/crypto_data.ods
INFO: Processing ALGO
ERROR: Fatal exception occurred:
Traceback (most recent call last):
  File "/home/james/.local/lib/python3.10/site-packages/rp2/rp2_main.py", line 93, in _rp2_main_internal
    input_data: InputData = parse_ods(configuration=configuration, asset=asset, input_file_handle=input_file_handle)
  File "/home/james/.local/lib/python3.10/site-packages/rp2/ods_parser.py", line 128, in parse_ods
    _create_and_process_transaction(
  File "/home/james/.local/lib/python3.10/site-packages/rp2/ods_parser.py", line 168, in _create_and_process_transaction
    transaction: AbstractTransaction = _create_transaction(configuration, current_table_type, internal_id, row_values)
  File "/home/james/.local/lib/python3.10/site-packages/rp2/ods_parser.py", line 277, in _create_transaction
    transaction = InTransaction(**argument_pack)
  File "/home/james/.local/lib/python3.10/site-packages/rp2/in_transaction.py", line 79, in __init__
    self.__fiat_in_no_fee = configuration.type_check_positive_decimal("fiat_in_no_fee", fiat_in_no_fee, non_zero=True)
  File "/home/james/.local/lib/python3.10/site-packages/rp2/configuration.py", line 279, in type_check_positive_decimal
    raise RP2ValueError(f"Parameter '{name}' has zero value")
rp2.rp2_error.RP2ValueError: Parameter 'fiat_in_no_fee' has zero value
INFO: Log file: ./log/rp2_2022_03_29_20_54_35_442552.log
INFO: Generated output directory: output/
INFO: Done

If I set "USD In" fields to Spot Price * Crypto In, it seems to process those fields correctly.

Not sure if this is still part of the same problem now, since Spot Price technically now shows up. I can close and cause separate issue if that makes sense.

eprbell commented 2 years ago

Sure, let's open a new issue. I think I know what the problem is: it's probably because Coinbase incorrectly reports the native_amount as 0 (same root cause as this issue) and DaLI uses that value to fill in the fiat_in fields. It should be easy to fix.

Before closing this issue: can you confirm the Algo spot price is between $0.4 and $3?

Thanks for providing all the data to help fix the bug!

jameskupke commented 2 years ago

Sure, here's the timestamps and spot prices from the ods file:

Timestamp Spot Price
2021-04-27 11:19:48 +0000 $1.28
2021-04-28 12:22:48 +0000 $1.27
2021-04-29 12:15:39 +0000 $1.37
2021-04-30 14:07:58 +0000 $1.40
2021-05-01 12:24:16 +0000 $1.39
2021-05-02 10:59:38 +0000 $1.37
2021-05-03 14:01:51 +0000 $1.43
2021-05-04 10:37:18 +0000 $1.44
2021-05-05 11:07:01 +0000 $1.39
2021-05-06 12:12:58 +0000 $1.56
2021-05-07 13:06:17 +0000 $1.64
2021-05-08 14:17:02 +0000 $1.54
2021-05-09 14:03:53 +0000 $1.49
2021-05-10 12:44:16 +0000 $1.48
2021-05-11 11:05:05 +0000 $1.42
2021-05-12 12:04:09 +0000 $1.42
2021-05-13 12:57:51 +0000 $1.36
2021-05-14 11:22:14 +0000 $1.40
2021-05-15 11:26:04 +0000 $1.42
2021-05-16 13:23:07 +0000 $1.48
2021-05-17 12:41:04 +0000 $1.37
2021-05-18 12:29:58 +0000 $1.38
2021-05-19 12:48:05 +0000 $0.96
2021-05-20 12:24:49 +0000 $1.07
2021-05-21 12:20:53 +0000 $1.08
2021-05-22 13:27:50 +0000 $0.98
2021-05-23 12:54:41 +0000 $0.79
2021-05-24 11:34:18 +0000 $0.96
2021-05-25 10:54:07 +0000 $0.90
2021-05-26 12:06:57 +0000 $1.05
2021-05-27 11:32:26 +0000 $1.01
2021-05-28 12:59:07 +0000 $0.90
2021-05-29 11:17:31 +0000 $0.86
2021-05-30 13:03:10 +0000 $0.87
2021-05-31 12:03:17 +0000 $0.90
2021-06-01 11:24:31 +0000 $0.91
2021-06-02 14:03:05 +0000 $0.94
2021-06-03 11:00:31 +0000 $1.07
2021-06-04 13:55:14 +0000 $1.03
2021-06-05 14:33:58 +0000 $1.00
2021-06-06 13:05:42 +0000 $1.01
2021-06-07 13:03:21 +0000 $1.09
2021-06-08 14:16:36 +0000 $0.89
2021-06-09 11:40:28 +0000 $1.09
2021-06-10 13:59:52 +0000 $1.01
2021-06-11 11:42:10 +0000 $1.02
2021-06-12 13:50:37 +0000 $0.98
2021-06-13 11:22:35 +0000 $1.02
2021-06-14 10:44:41 +0000 $1.04
2021-06-15 11:41:35 +0000 $1.05
2021-06-16 13:32:37 +0000 $1.01
2021-06-17 13:41:44 +0000 $1.06
2021-06-18 10:42:12 +0000 $1.03
2021-06-19 14:04:54 +0000 $0.96
2021-06-20 13:22:57 +0000 $0.90
2021-06-21 13:20:18 +0000 $0.84
2021-06-22 12:09:02 +0000 $0.72
2021-06-23 11:43:23 +0000 $0.84
2021-06-24 13:52:50 +0000 $0.86
2021-06-25 11:59:26 +0000 $0.83
2021-06-26 12:51:35 +0000 $0.80
2021-06-27 14:23:04 +0000 $0.79
2021-06-28 14:00:12 +0000 $0.83
2021-06-29 13:59:17 +0000 $0.92
2021-06-30 11:38:16 +0000 $0.86
2021-07-01 12:26:56 +0000 $0.84
2021-07-02 12:30:26 +0000 $0.81
2021-07-03 11:35:48 +0000 $0.86
2021-07-04 13:45:53 +0000 $0.90
2021-07-05 14:32:54 +0000 $0.86
2021-07-06 12:57:37 +0000 $0.93
2021-07-07 13:25:32 +0000 $0.93
2021-07-08 12:43:31 +0000 $0.87
2021-07-09 14:43:02 +0000 $0.92
2021-07-10 11:16:17 +0000 $0.88
2021-07-11 12:54:12 +0000 $0.91
2021-07-12 14:19:44 +0000 $0.88
2021-07-13 13:43:56 +0000 $0.86
2021-07-14 12:59:00 +0000 $0.85
2021-07-15 14:29:54 +0000 $0.83
2021-07-16 13:13:17 +0000 $0.80
2021-07-17 11:50:51 +0000 $0.78
2021-07-18 13:33:44 +0000 $0.80
2021-07-19 13:14:14 +0000 $0.76
2021-07-20 13:49:30 +0000 $0.69
2021-07-21 13:17:58 +0000 $0.75
2021-07-22 18:42:14 +0000 $0.79
2021-07-23 15:22:51 +0000 $0.80
2021-07-24 15:00:54 +0000 $0.84
2021-07-25 17:44:25 +0000 $0.87
2021-07-26 11:54:28 +0000 $0.90
2021-07-27 17:58:10 +0000 $0.83
2021-07-28 18:19:44 +0000 $0.84
2021-07-29 15:14:12 +0000 $0.84
2021-07-30 17:17:24 +0000 $0.83
2021-07-31 17:19:15 +0000 $0.84
2021-08-01 17:39:26 +0000 $0.84
2021-08-02 16:43:20 +0000 $0.84
2021-08-03 14:57:18 +0000 $0.83
2021-08-04 15:06:41 +0000 $0.83
2021-08-05 14:07:54 +0000 $0.84
2021-08-06 13:38:52 +0000 $0.85
2021-08-07 13:09:26 +0000 $0.86
2021-08-08 14:07:17 +0000 $0.88
2021-08-09 16:21:17 +0000 $0.87
2021-08-10 12:25:52 +0000 $0.90
2021-08-11 13:29:58 +0000 $0.93
2021-08-12 12:25:35 +0000 $0.90
2021-08-13 16:27:40 +0000 $0.92
2021-08-14 12:24:21 +0000 $0.93
2021-08-15 13:44:21 +0000 $0.93
2021-08-16 12:36:56 +0000 $1.00
2021-08-17 14:30:36 +0000 $0.96
2021-08-18 12:51:31 +0000 $0.90
2021-08-19 11:05:16 +0000 $0.98
2021-08-20 13:52:44 +0000 $1.11
2021-08-21 14:28:13 +0000 $1.20
2021-08-22 15:00:28 +0000 $1.14
2021-08-23 13:42:31 +0000 $1.17
2021-08-24 17:42:13 +0000 $1.06
2021-08-25 11:35:11 +0000 $1.02
2021-08-26 16:06:48 +0000 $1.00
2021-08-27 12:25:52 +0000 $1.02
2021-08-28 11:15:05 +0000 $1.04
2021-08-29 13:36:01 +0000 $1.02
2021-08-30 13:21:57 +0000 $0.99
2021-08-31 18:08:52 +0000 $1.12
2021-09-01 17:31:58 +0000 $1.15
2021-09-02 17:12:58 +0000 $1.14
2021-09-03 15:19:40 +0000 $1.24
2021-09-04 12:26:40 +0000 $1.22
2021-09-05 14:48:29 +0000 $1.29
2021-09-06 13:52:59 +0000 $1.34
2021-09-07 11:42:26 +0000 $1.36
2021-09-08 15:33:28 +0000 $1.74
2021-09-09 15:39:42 +0000 $2.24
2021-09-10 13:04:01 +0000 $2.05
2021-09-11 18:43:02 +0000 $2.08
2021-09-12 15:18:12 +0000 $2.33
2021-09-13 16:47:12 +0000 $2.26
2021-09-14 17:54:29 +0000 $2.07
2021-09-15 17:29:59 +0000 $2.06
2021-09-16 11:44:49 +0000 $2.20
2021-09-17 18:32:00 +0000 $1.94
2021-09-18 15:30:13 +0000 $2.04
2021-09-19 15:07:00 +0000 $2.03
2021-09-21 11:55:13 +0000 $1.82
2021-09-22 14:49:44 +0000 $1.78
2021-09-23 15:56:31 +0000 $1.97
2021-09-24 09:21:34 +0000 $1.80
2021-09-24 11:15:38 +0000 $1.71
2021-09-25 13:53:23 +0000 $1.77
2021-09-26 17:17:24 +0000 $1.71
2021-09-27 14:55:27 +0000 $1.78
2021-09-28 14:23:26 +0000 $1.63
2021-09-29 13:50:07 +0000 $1.65
2021-09-30 17:59:50 +0000 $1.62
2021-10-01 16:09:29 +0000 $1.76
2021-10-02 16:42:03 +0000 $1.83
2021-10-03 10:48:18 +0000 $1.80
2021-10-04 12:29:49 +0000 $1.97
2021-10-05 15:46:49 +0000 $1.94
2021-10-06 17:17:14 +0000 $1.90
2021-10-07 11:03:25 +0000 $1.83
2021-10-08 12:51:02 +0000 $1.91
2021-10-09 17:42:40 +0000 $1.90
2021-10-10 10:52:34 +0000 $1.86
2021-10-11 17:21:59 +0000 $1.77
2021-10-12 12:27:10 +0000 $1.63
2021-10-13 12:10:55 +0000 $1.73
2021-10-14 14:43:14 +0000 $1.88
2021-10-15 11:49:18 +0000 $1.78
2021-10-16 13:58:29 +0000 $1.81
2021-10-17 18:42:34 +0000 $1.81
2021-10-18 12:18:44 +0000 $1.75
2021-10-19 16:58:35 +0000 $1.72
2021-10-20 18:43:57 +0000 $1.82
2021-10-21 12:13:30 +0000 $1.81
2021-10-22 13:10:59 +0000 $1.91
2021-10-23 14:08:33 +0000 $1.90
2021-10-24 11:29:10 +0000 $1.86
2021-10-25 17:28:48 +0000 $1.96
2021-10-26 14:21:19 +0000 $1.98
2021-10-27 11:38:30 +0000 $1.82
2021-10-28 13:40:20 +0000 $1.85
2021-10-29 15:04:42 +0000 $1.85
2021-10-30 14:31:53 +0000 $1.81
2021-10-31 17:30:42 +0000 $1.84
2021-11-01 10:52:12 +0000 $1.84
2021-11-02 17:10:53 +0000 $1.89
2021-11-03 16:19:50 +0000 $1.91
2021-11-04 15:40:22 +0000 $1.86

Looks okay to close?

eprbell commented 2 years ago

Looks good: let's close this, thanks.

eprbell commented 2 years ago

The USD In set to 0 was caused by Coinbase returning low-precision fiat numbers (and rounding to 0 any amount < $0.01). I added a workaround in the latest version. @jameskupke, could you test it and let me know if it works? Thanks!

jameskupke commented 2 years ago

Ok, running on 0.4.0