search

eprbell / dali-rp2

DaLI (Data Loader Interface) is a data loader and input generator for RP2 (https://pypi.org/project/rp2), the privacy-focused, free, open-source cryptocurrency tax calculator: DaLI removes the need to manually prepare RP2 input files. Just like RP2, DaLI is also free, open-source and it prioritizes user privacy.
https://pypi.org/project/dali-rp2/
Apache License 2.0
63 stars 42 forks source link

Differentiate None and Empty handling for optional fields #183

Closed topherbuckley closed 1 year ago

topherbuckley commented 1 year ago

As per this

Note that pytests fail here. 

=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.10.6, pytest-7.3.1, pluggy-1.0.0 -- /media/christopher/HDD/git/dali-rp2/.venv/bin/python
cachedir: .pytest_cache
rootdir: /media/christopher/HDD/git/dali-rp2
plugins: mock-3.10.0
collected 61 items                                                                                                                                                                                        

tests/test_abstract_pair_converter.py::TestAbstractPairConverterPlugin::test_mapped_graph_class PASSED                                                                                              [  1%]
tests/test_cache.py::TestCache::test_dict_cache PASSED                                                                                                                                              [  3%]
tests/test_cache.py::TestCache::test_list_cache PASSED                                                                                                                                              [  4%]
tests/test_historical_bar.py::test_historical_bar_is_frozen PASSED                                                                                                                                  [  6%]
tests/test_historical_bar.py::test_derive_transaction_price_invalid PASSED                                                                                                                          [  8%]
tests/test_historical_bar.py::test_derive_transaction_price_open PASSED                                                                                                                             [  9%]
tests/test_historical_bar.py::test_derive_transaction_price_high PASSED                                                                                                                             [ 11%]
tests/test_historical_bar.py::test_derive_transaction_price_low PASSED                                                                                                                              [ 13%]
tests/test_historical_bar.py::test_derive_transaction_price_close PASSED                                                                                                                            [ 14%]
tests/test_historical_bar.py::test_derive_transaction_price_nearest_open[-1] PASSED                                                                                                                 [ 16%]
tests/test_historical_bar.py::test_derive_transaction_price_nearest_open[0] PASSED                                                                                                                  [ 18%]
tests/test_historical_bar.py::test_derive_transaction_price_nearest_open[1] PASSED                                                                                                                  [ 19%]
tests/test_historical_bar.py::test_derive_transaction_price_nearest_open[29] PASSED                                                                                                                 [ 21%]
tests/test_historical_bar.py::test_derive_transaction_price_nearest_close[31] PASSED                                                                                                                [ 22%]
tests/test_historical_bar.py::test_derive_transaction_price_nearest_close[59] PASSED                                                                                                                [ 24%]
tests/test_historical_bar.py::test_derive_transaction_price_nearest_close[60] PASSED                                                                                                                [ 26%]
tests/test_historical_bar.py::test_derive_transaction_price_nearest_close[61] PASSED                                                                                                                [ 27%]
tests/test_ods_output_diff.py::TestODSOutputDiff::test_crypto_data_ods ERROR                                                                                                                        [ 29%]
tests/test_ods_output_diff.py::TestODSOutputDiff::test_fifo_rp2_full_report_ods ERROR                                                                                                               [ 31%]
tests/test_ods_output_diff.py::TestODSOutputDiff::test_fifo_tax_report_us_ods ERROR                                                                                                                 [ 32%]
tests/test_ods_output_diff.py::TestODSOutputDiff::test_ods_sheet_size ERROR                                                                                                                         [ 34%]
tests/test_plugin_binance_com.py::TestBinance::test_deposits PASSED                                                                                                                                 [ 36%]
tests/test_plugin_binance_com.py::TestBinance::test_trades PASSED                                                                                                                                   [ 37%]
tests/test_plugin_binance_com.py::TestBinance::test_gains PASSED                                                                                                                                    [ 39%]
tests/test_plugin_binance_com.py::TestBinance::test_implicit_api PASSED                                                                                                                             [ 40%]
tests/test_plugin_binance_com.py::TestBinance::test_withdrawals PASSED                                                                                                                              [ 42%]
tests/test_plugin_binance_com_supplemental_csv.py::TestBinanceCsv::test_autoinvest PASSED                                                                                                           [ 44%]
tests/test_plugin_binance_com_supplemental_csv.py::TestBinanceCsv::test_betheth PASSED                                                                                                              [ 45%]
tests/test_plugin_bitbank_supplemental_csv.py::TestBitbank::test_withdrawals PASSED                                                                                                                 [ 47%]
tests/test_plugin_ccxt.py::TestCcxtPlugin::test_unknown_exchange PASSED                                                                                                                             [ 49%]
tests/test_plugin_ccxt.py::TestCcxtPlugin::test_historical_prices PASSED                                                                                                                            [ 50%]
tests/test_plugin_ccxt.py::TestCcxtPlugin::test_missing_historical_prices PASSED                                                                                                                    [ 52%]
tests/test_plugin_ccxt.py::TestCcxtPlugin::test_missing_fiat_pair PASSED                                                                                                                            [ 54%]
tests/test_plugin_ccxt.py::TestCcxtPlugin::test_no_fiat_pair PASSED                                                                                                                                 [ 55%]
tests/test_plugin_ccxt.py::TestCcxtPlugin::test_nonusd_fiat_pair PASSED                                                                                                                             [ 57%]
tests/test_plugin_ccxt.py::TestCcxtPlugin::test_fiat_pair PASSED                                                                                                                                    [ 59%]
tests/test_plugin_ccxt.py::TestCcxtPlugin::test_kraken_csv PASSED                                                                                                                                   [ 60%]
tests/test_plugin_ccxt.py::TestCcxtPlugin::test_locked_exchange PASSED                                                                                                                              [ 62%]
tests/test_plugin_coinbase.py::TestTrade::test_eth2_stake PASSED                                                                                                                                    [ 63%]
tests/test_plugin_coinbase_pro.py::TestSwapFill::test_buy_side PASSED                                                                                                                               [ 65%]
tests/test_plugin_coincheck_supplemental.py::TestCoincheck::test_trades PASSED                                                                                                                      [ 67%]
tests/test_plugin_historic_crypto.py::TestHistoricCryptoPlugin::test_historical_prices PASSED                                                                                                       [ 68%]
tests/test_plugin_historic_crypto.py::TestHistoricCryptoPlugin::test_missing_historical_prices PASSED                                                                                               [ 70%]
tests/test_plugin_kraken.py::test_kraken PASSED                                                                                                                                                     [ 72%]
tests/test_plugin_kraken_csv_download.py::TestKrakenCsvDownload::test_chunking PASSED                                                                                                               [ 73%]
tests/test_plugin_ods_rp2_input.py::TestRP2InputOds::test_all_transactions_found PASSED                                                                                                             [ 75%]
tests/test_plugin_ods_rp2_input.py::TestRP2InputOds::test_in_transactions PASSED                                                                                                                    [ 77%]
tests/test_plugin_ods_rp2_input.py::TestRP2InputOds::test_intra_transactions PASSED                                                                                                                 [ 78%]
tests/test_plugin_ods_rp2_input.py::TestRP2InputOds::test_out_transactions PASSED                                                                                                                   [ 80%]
tests/test_plugin_pionex.py::TestPionex::test_trades PASSED                                                                                                                                         [ 81%]
tests/test_plugin_pionex.py::TestPionex::test_transfers PASSED                                                                                                                                      [ 83%]
tests/test_transaction_resolver.py::test_resolve_intra_intra_transaction[None-None-None-None] FAILED                                                                                                [ 85%]
tests/test_transaction_resolver.py::test_resolve_intra_intra_transaction[None-xxx-None-xxx;] PASSED                                                                                                 [ 86%]
tests/test_transaction_resolver.py::test_resolve_intra_intra_transaction[None-None-yyy-yyy;] PASSED                                                                                                 [ 88%]
tests/test_transaction_resolver.py::test_resolve_intra_intra_transaction[None-xxx-yyy-xxx; yyy;] PASSED                                                                                             [ 90%]
tests/test_transaction_resolver.py::test_resolve_intra_intra_transaction[None-xxx-xxx-xxx;] PASSED                                                                                                  [ 91%]
tests/test_transaction_resolver.py::test_resolve_intra_intra_transaction[prior-None-None-prior;] PASSED                                                                                             [ 93%]
tests/test_transaction_resolver.py::test_resolve_intra_intra_transaction[prior-xxx-None-prior; xxx;] PASSED                                                                                         [ 95%]
tests/test_transaction_resolver.py::test_resolve_intra_intra_transaction[prior-None-yyy-prior; yyy;] PASSED                                                                                         [ 96%]
tests/test_transaction_resolver.py::test_resolve_intra_intra_transaction[prior-xxx-yyy-prior; xxx; yyy;] PASSED                                                                                     [ 98%]
tests/test_transaction_resolver.py::test_resolve_intra_intra_transaction[prior-xxx-xxx-prior; xxx;] PASSED                                                                                          [100%]

================================================================================================= ERRORS ==================================================================================================
________________________________________________________________________ ERROR at setup of TestODSOutputDiff.test_crypto_data_ods _________________________________________________________________________
Traceback (most recent call last):
  File "/media/christopher/HDD/git/dali-rp2/tests/test_ods_output_diff.py", line 45, in setUpClass
    cls._generate(cls.output_dir, "test", "test_config")
  File "/media/christopher/HDD/git/dali-rp2/tests/test_ods_output_diff.py", line 82, in _generate
    run(arguments, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['rp2_us', '-m', 'fifo', '-o', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff', '-p', 'test_', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff/test_crypto_data.ini', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff/test_crypto_data.ods']' returned non-zero exit status 1.
------------------------------------------------------------------------------------------ Captured stdout setup ------------------------------------------------------------------------------------------
Configuration file '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff/test_crypto_data.ini' not found
usage: rp2_us [-h] [-a ASSET] [-f DATE] [-g GENERATION_LANGUAGE] [-m METHOD]
              [-o OUTPUT_DIR] [-p PREFIX] [-t DATE] [-v]
              CONFIGURATION INPUT

Generate capital gain/loss report and balances for crypto holdings. Links:
- documentation: https://github.com/eprbell/rp2/blob/main/README.md
- FAQ: https://github.com/eprbell/rp2/blob/main/docs/user_faq.md
- support RP2 by leaving a star on Github: https://github.com/eprbell/rp2

positional arguments:
  CONFIGURATION         Configuration file
  INPUT                 ODS file containing input transactions

options:
  -h, --help            show this help message and exit
  -a ASSET, --asset ASSET
                        Generate report only for the given ASSET
  -f DATE, --from_date DATE
                        Generate report from the given date (in ISO 8601 format: e.g. YYYY-MM-DD)
  -g GENERATION_LANGUAGE, --generation-language GENERATION_LANGUAGE
                        Language to use during generation (in ISO 639-1 format)
  -m METHOD, --method METHOD
                        accounting method (default: ''). Supported values: fifo
  -o OUTPUT_DIR, --output_dir OUTPUT_DIR
                        Write output to OUTPUT_DIR  (default: 'output/')
  -p PREFIX, --prefix PREFIX
                        Prepend output file names with PREFIX
  -t DATE, --to_date DATE
                        Generate report up to the given date (in ISO 8601 format: e.g. YYYY-MM-DD)
  -v, --version         Print RP2 version
------------------------------------------------------------------------------------------ Captured stderr setup ------------------------------------------------------------------------------------------
INFO: Country: us
INFO: Initialized input plugin 'dali.plugin.input.csv.manual'
INFO: Initialized input plugin 'dali.plugin.input.csv.trezor Bob BTC'
INFO: Initialized input plugin 'dali.plugin.input.csv.trezor Alice BTC'
INFO: No pair converter plugins found in configuration file: using default pair converters.
INFO: Reading crypto data using plugin 'dali.plugin.input.csv.manual'
INFO: Reading crypto data using plugin 'dali.plugin.input.csv.trezor Bob BTC'
INFO: Reading crypto data using plugin 'dali.plugin.input.csv.trezor Alice BTC'
ERROR: Fatal exception occurred:
Traceback (most recent call last):
  File "/media/christopher/HDD/git/dali-rp2/src/dali/dali_main.py", line 176, in _dali_main_internal
    result_list = pool.map(_input_plugin_helper, input_plugin_args_list)
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/media/christopher/HDD/git/dali-rp2/src/dali/dali_main.py", line 214, in _input_plugin_helper
    plugin_transactions = input_plugin.load(country)
  File "/media/christopher/HDD/git/dali-rp2/src/dali/plugin/input/csv/manual.py", line 99, in load
    self._load_in_file(result)
  File "/media/christopher/HDD/git/dali-rp2/src/dali/plugin/input/csv/manual.py", line 133, in _load_in_file
    InTransaction(
  File "/media/christopher/HDD/git/dali-rp2/src/dali/in_transaction.py", line 82, in __init__
    raise RP2RuntimeError(
rp2.rp2_error.RP2RuntimeError: Internal error: both 'crypto_fee' and 'fiat_fee' are defined, instead of only one: their values are  and 10 respectively
INFO: Log file: ./log/rp2_2023_05_27_20_07_14_555381.log
INFO: Generated output directory: /media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff
INFO: Done
____________________________________________________________________ ERROR at setup of TestODSOutputDiff.test_fifo_rp2_full_report_ods ____________________________________________________________________
Traceback (most recent call last):
  File "/media/christopher/HDD/git/dali-rp2/tests/test_ods_output_diff.py", line 45, in setUpClass
    cls._generate(cls.output_dir, "test", "test_config")
  File "/media/christopher/HDD/git/dali-rp2/tests/test_ods_output_diff.py", line 82, in _generate
    run(arguments, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['rp2_us', '-m', 'fifo', '-o', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff', '-p', 'test_', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff/test_crypto_data.ini', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff/test_crypto_data.ods']' returned non-zero exit status 1.
_____________________________________________________________________ ERROR at setup of TestODSOutputDiff.test_fifo_tax_report_us_ods _____________________________________________________________________
Traceback (most recent call last):
  File "/media/christopher/HDD/git/dali-rp2/tests/test_ods_output_diff.py", line 45, in setUpClass
    cls._generate(cls.output_dir, "test", "test_config")
  File "/media/christopher/HDD/git/dali-rp2/tests/test_ods_output_diff.py", line 82, in _generate
    run(arguments, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['rp2_us', '-m', 'fifo', '-o', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff', '-p', 'test_', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff/test_crypto_data.ini', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff/test_crypto_data.ods']' returned non-zero exit status 1.
_________________________________________________________________________ ERROR at setup of TestODSOutputDiff.test_ods_sheet_size _________________________________________________________________________
Traceback (most recent call last):
  File "/media/christopher/HDD/git/dali-rp2/tests/test_ods_output_diff.py", line 45, in setUpClass
    cls._generate(cls.output_dir, "test", "test_config")
  File "/media/christopher/HDD/git/dali-rp2/tests/test_ods_output_diff.py", line 82, in _generate
    run(arguments, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['rp2_us', '-m', 'fifo', '-o', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff', '-p', 'test_', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff/test_crypto_data.ini', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff/test_crypto_data.ods']' returned non-zero exit status 1.
================================================================================================ FAILURES =================================================================================================
________________________________________________________________________ test_resolve_intra_intra_transaction[None-None-None-None] ________________________________________________________________________
Traceback (most recent call last):
  File "/media/christopher/HDD/git/dali-rp2/tests/test_transaction_resolver.py", line 86, in test_resolve_intra_intra_transaction
    assert resolved.notes == resolved_notes
AssertionError: assert '' == None
 +  where '' = IntraTransaction(plugin='DaLI Resolver', unique_id='unique_id', raw_data='raw_data1\nraw_data2', timestamp='2022-01-01 00:00:00+0000', asset='asset', from_exchange='from_exchange1', from_holder='from_holder1', to_exchange='to_exchange2', to_holder='to_holder2', spot_price=1000.0, crypto_sent=1.0, crypto_received=1.0, notes=).notes
========================================================================================= short test summary info =========================================================================================
FAILED tests/test_transaction_resolver.py::test_resolve_intra_intra_transaction[None-None-None-None] - AssertionError: assert '' == None
ERROR tests/test_ods_output_diff.py::TestODSOutputDiff::test_crypto_data_ods - subprocess.CalledProcessError: Command '['rp2_us', '-m', 'fifo', '-o', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff', '-p', 'test_', '/media/christopher/HDD/git/dali-rp2/output/t...
ERROR tests/test_ods_output_diff.py::TestODSOutputDiff::test_fifo_rp2_full_report_ods - subprocess.CalledProcessError: Command '['rp2_us', '-m', 'fifo', '-o', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff', '-p', 'test_', '/media/christopher/HDD/git/dali-rp2/output/t...
ERROR tests/test_ods_output_diff.py::TestODSOutputDiff::test_fifo_tax_report_us_ods - subprocess.CalledProcessError: Command '['rp2_us', '-m', 'fifo', '-o', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff', '-p', 'test_', '/media/christopher/HDD/git/dali-rp2/output/t...
ERROR tests/test_ods_output_diff.py::TestODSOutputDiff::test_ods_sheet_size - subprocess.CalledProcessError: Command '['rp2_us', '-m', 'fifo', '-o', '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff', '-p', 'test_', '/media/christopher/HDD/git/dali-rp2/output/t...
================================================================================= 1 failed, 56 passed, 4 errors in 19.69s =================================================================================

I might not have a ton of time coming up here, so feel free to close this PR if there is no quick fix in sight.

macanudo527 commented 1 year ago

According to the log, It seems like you are missing a configuration file:

Configuration file '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff/test_crypto_data.ini' not found
topherbuckley commented 1 year ago

According to the log, It seems like you are missing a configuration file:

Configuration file '/media/christopher/HDD/git/dali-rp2/output/test_ods_output_diff/test_crypto_data.ini' not found

Nope, no such file on my local working directory. I'm not showing a diff in git diff from the current main branch of this repo. Is this a file you have or that is supposed to be generated during the test perhaps?

eprbell commented 1 year ago

This means there are probably other considerations to the validation of optional fields and it's not as easy as our previous analysis suggested: unfortunately I don't have time to dig into this. In any event this is a corner case that doesn't have any real-world implications that I'm aware of.

topherbuckley commented 1 year ago

This means there are probably other considerations to the validation of optional fields and it's not as easy as our previous analysis suggested: unfortunately I don't have time to dig into this. In any event this is a corner case that doesn't have any real-world implications that I'm aware of.

Alrighty. I'll continue to discuss this over here with you when you have time, but for now I'll close this PR and I'll revisit this after gaining a better understanding of it all.