Coinbase Pro REST plugin: transfer across portfolios not yet supported

[gbtorrance]

Hi. I'm just getting started with Dali-RP2. I'm trying to use the Coinbase Pro plugin, but I'm getting the following exception. Any guidance on how to resolve? Thanks!

---

2022-09-09 12:49:38,159/dali/INFO: Country: us 2022-09-09 12:49:38,171/dali/INFO: Initialized input plugin 'dali.plugin.input.rest.coinbase_pro ' 2022-09-09 12:49:38,173/dali/INFO: No pair converter plugins found in configuration file: using default pair converters. 2022-09-09 12:49:38,178/dali/INFO: Reading crypto data using plugin 'dali.plugin.input.rest.coinbase_pro ' 2022-09-09 12:50:01,429/dali/ERROR: Fatal exception occurred: Traceback (most recent call last): File "/home/mint21/.local/lib/python3.10/site-packages/dali/dali_main.py", line 163, in _dali_main_internal result_list = pool.map(_input_plugin_helper, input_plugin_args_list) File "/usr/lib/python3.10/multiprocessing/pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/lib/python3.10/multiprocessing/pool.py", line 771, in get raise self._value File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, kwds)) File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar return list(map(args)) File "/home/mint21/.local/lib/python3.10/site-packages/dali/dali_main.py", line 199, in _input_plugin_helper plugin_transactions = input_plugin.load() File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 157, in load process_account_result_list = pool.map(self._process_account, accounts) File "/usr/lib/python3.10/multiprocessing/pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/lib/python3.10/multiprocessing/pool.py", line 771, in get raise self._value File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(args, kwds)) File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar return list(map(*args)) File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 185, in _process_account self._process_transfer(transaction, currency, intra_transaction_list) File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 224, in _process_transfer transfer_id: str = transaction_details[_TRANSFER_ID] KeyError: 'transfer_id' 2022-09-09 12:50:01,434/dali/INFO: Log file: ./log/rp2_2022_09_09_12_49_37_655412.log 2022-09-09 12:50:01,434/dali/INFO: Generated output directory: output 2022-09-09 12:50:01,434/dali/INFO: Done

[eprbell]

Thanks for reporting a bug! To resolve it I'll need the debug log line of the problematic transaction. Here's what to do:

• first read https://github.com/eprbell/dali-rp2/blob/main/CONTRIBUTING.md. What we need here is one single line from your debug log, but this line may contain personal information: make sure you're OK with sharing it, or anonymize it before sharing it;
• in your configuration.ini file delete any data loader plugin section except the Coinbase Pro one: we want to run only the CBPro data loader;
• in the CBPro section of the configuration.ini file, make sure to pass thread_count=1 (if you need the reference manual: https://github.com/eprbell/dali-rp2/blob/main/docs/configuration_file.md#coinbase-pro-section-rest);
• delete the log dir;
• run rp2 with debug log level: LOG_LEVEL=DEBUG bin/dali_us -t 1 -s -o output/ my_config.ini;
• the log directory contains the new debug-enabled log file: look inside the log file for the last entry that was read from Coinbase Pro. That line should be the one we need. The error is saying it's a transfer transaction without a transfer id: both strange and interesting. Hopefully this log line will shed some light on this.

[gbtorrance]

Thanks for the reply and the info! I'm including the last few lines of the log (prior to the traceback), as they seem possibly related (and therefore relevant).

2022-09-09 15:13:11,863/Coinbase Pro/Greg/DEBUG: Transfer: {"id": "58a11155-fe96-4f7b-a915-10fd151212ae", "type": "withdraw", "created_at": "2021-10-09 15:15:13.121313+00", "completed_at": "2021-10-09 15:15:47.587375+00", "account_id": "6c7f97c7-4deb-497f-9f02-99d5b91038db", "user_id": "60d8c47a1c22150e7ccc6df4", "amount": "12.82100000", "details": {"fee": "0.000005", "subtotal": "12.820995", "sent_to_address": "4AJisbFMLuxaEstFqLBiF1PM2MBWzyzzUKPjsp6FK1mb", "coinbase_account_id": "eeec9642-925f-5611-b485-40c5e3165087", "coinbase_withdrawal_id": "6161b2030b25fc02cbb337b9", "coinbase_transaction_id": "6161b20140f4bf00015b0702", "crypto_transaction_hash": "3hZGYkHoYFNYYPCL1eXbsa3sgrAPYzcmUZbuva4Bm43ZQ7rzhkPo7233KYwqAYnJ8H4P23nNUuVHST6s4Rx2cvd7", "coinbase_payment_method_id": ""}, "canceled_at": null, "processed_at": "2021-10-09 15:15:47.596282+00", "user_nonce": "1633792455923", "idem": null} 2022-09-09 15:13:11,863/Coinbase Pro/Greg/DEBUG: Transfer is a Coinbase transaction already captured by Coinbase plugin: ignoring. 2022-09-09 15:13:11,864/Coinbase Pro/Greg/DEBUG: Transaction: {"id": "11275304484", "amount": "12.8210000000000000", "balance": "12.8210000000000000", "created_at": "2021-10-09T15:11:55.187005Z", "type": "transfer", "details": {"from": "ccbef35f-d0c3-4a80-8ec5-f7e701f58ea1", "profile_transfer_id": "19d52363-3563-4398-8e6f-66c039b1bcff", "to": "6c7f97c7-4deb-497f-9f02-99d5b91038db"}} 2022-09-09 15:13:11,867/dali/ERROR: Fatal exception occurred:

[eprbell]

Yes, the last line fits the bill:

• "type": "transfer"
• no "transfer_id" in the "details" section

I haven't seen this type of transaction before: could you look for it on Coinbase Pro and share more about the nature of this transaction? What kind of momevent is it capturing?

Some notes:

• while there is no "transfer_id", there is a "profile_transfer_id"
• there are two "from" and "to" fields that seem to have Coinbase ids: is this a transfer from / to Coinbase?

[eprbell]

I suspect this is a transfer to/from Coinbase, which is denoted by a different keyword: profile_transfer_id rather than transfer_id.

Could you check the following:

• does this transaction appear in the coinbase debug log as well? To find out: use a .ini file containing only the coinbase data loader and run dali with debug log level as described above. If you find a transaction in the log that matches the amounts and other details of this one, then this is another occurrence of this: https://github.com/eprbell/dali-rp2/blob/main/src/dali/plugin/input/rest/coinbase_pro.py#L232
• log on the websites (both Coinbase and Coinbase Pro), look for this transaction and describe here what it is and what it does.

[gbtorrance]

I did some digging. It's not a transfer to/from Coinbase and Coinbase Pro, but it is between two portfolios within Coinbase Pro. What seems to have happened is I transferred SOL from my Secondary Portfolio to my Primary Portfolio. The Secondary Portfolio shows a single withdrawal transaction, and the Primary shows a deposit and a withdrawal. Here's the info:

Primary Portfolio

Withdraw

11:15:47 am - Oct 09, 2021 EDT -12.821000000 0.000000000 Transfer ID: 58a11155-fe96-4f7b-a915-10fd151212ae

Deposit

11:11:55 am - Oct 09, 2021 EDT 12.821000000 12.821000000 $445.27 from: ccbef35f-d0c3-4a80-8ec5-f7e701f58ea1 profileTransferId: 19d52363-3563-4398-8e6f-66c039b1bcff to: 6c7f97c7-4deb-497f-9f02-99d5b91038db

Secondary Portfolio

Withdraw

11:11:55 am - Oct 09, 2021 EDT -12.821000000 0.000000000 from: ccbef35f-d0c3-4a80-8ec5-f7e701f58ea1 profileTransferId: 19d52363-3563-4398-8e6f-66c039b1bcff to: 6c7f97c7-4deb-497f-9f02-99d5b91038db

[eprbell]

Interesting. Can you also clarify what is a primary vs secondary cbpro account? I'm not familiar with that. How many cbpro sections do you have in your .ini file? Does the secondary account have a different secret/passphrase or the same as the primary one?

[gbtorrance]

I probably misled you a bit by saying Primary and Secondary. For me they are "primary" and "secondary", but you can have as many "portfolios" in Coinbase Pro as you like (as far as I am aware). It's basically a way of segregating a single account/login into separate "sub-accounts", where you can trade independently in each, and do transfers between portfolios. My "primary" I used for regular trading, and my "secondary" I used for bot trading with 3Commas. And in this particular case I transferred a SOL position from the secondary/3Commas/bot portfolio to the primary portfolio. Hopefully that makes sense.

As for the .ini file: That's a good point. At the moment I only have a data loader for the "primary". (Each portfolio does require separate keys for API access.)

[eprbell]

Thanks for the clarification. I think we need to run a few experiments since the multiple porfolio scenario is something I don't have and can't test directly.

Some more questions first:

• the run you did is for your secondary portfolio, correct?
• the withdrawal information is repeated both in your primary and secondary (but the deposit is only in the primary), right?

Here are the experiments to run.

Experiment 1:

• run DaLI again but this time with your primary account. Does it work or do you get any errors?
• if it works then look for the withdrawal and deposit lines and paste them here

Experiment 2:

• apply this diff:

  
  diff --git a/src/dali/plugin/input/rest/coinbase_pro.py b/src/dali/plugin/input/rest/coinbase_pro.py
  index a39b517..e541c13 100644
  --- a/src/dali/plugin/input/rest/coinbase_pro.py
  +++ b/src/dali/plugin/input/rest/coinbase_pro.py
  @@ -59,6 +59,7 @@ _MATCH: str = "match"
  _ORDER_ID: str = "order_id"
  _PRICE: str = "price"
  _PRODUCT_ID: str = "product_id"
  +_PROFILE_TRANSFER_ID: str = "profile_transfer_id"
  _SELL: str = "sell"
  _SIDE: str = "side"
  _SIZE: str = "size"
  @@ -221,7 +222,11 @@ class InputPlugin(AbstractInputPlugin):
       if amount < ZERO:
           amount = -amount  # type: ignore
       transaction_details: Any = transaction[_DETAILS]

• transfer_id: str = transaction_details[_TRANSFER_ID]
• transfer_id: str
• if _TRANSFER_ID in transaction_details:
• transfer_id = transaction_details[_TRANSFER_ID]
• elif _PROFILE_TRANSFER_ID in transaction_details:
• transfer_id = transaction_details[_PROFILE_TRANSFER_ID] transfer: Any = self.__get_transfer(transfer_id) transfer_details: Any = transfer[_DETAILS] crypto_hash: str = Keyword.UNKNOWN.value

• run DaLI again on the secondary portfolio. Does it go further? If it doesn't crash look for the line in the log immediately after the culplrit one (it should be a line starting with "Transfer:").

[gbtorrance]

Actually, the run I did initially was for my primary portfolio. Below is the info for my secondary ("bot") portfolio.

As you can see, there is an error for the secondary, also; and it is for the same SOL transaction as in the primary.

"the withdrawal information is repeated both in your primary and secondary (but the deposit is only in the primary), right?"

Right!

Regarding the Experiment 2, I'll probably need some guidance on how to do that. I'm currently using the installed software. I can clone the repo locally and can apply the diff, but can you tell me how to run it from the source? (I have no Python experience.)

---

2022-09-10 09:30:51,389/Coinbase Pro/Greg/DEBUG: Transaction: {"id": "11275304483", "amount": "-12.8210000000000000", "balance": "0.0000000000000000", "created_at": "2021-10-09T15:11:55.187005Z", "type": "transfer", "details": {"from": "ccbef35f-d0c3-4a80-8ec5-f7e701f58ea1", "profile_transfer_id": "19d52363-3563-4398-8e6f-66c039b1bcff", "to": "6c7f97c7-4deb-497f-9f02-99d5b91038db"}} 2022-09-10 09:30:51,392/dali/ERROR: Fatal exception occurred: Traceback (most recent call last): File "/home/mint21/.local/lib/python3.10/site-packages/dali/dali_main.py", line 163, in _dali_main_internal result_list = pool.map(_input_plugin_helper, input_plugin_args_list) File "/usr/lib/python3.10/multiprocessing/pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/lib/python3.10/multiprocessing/pool.py", line 771, in get raise self._value File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, kwds)) File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar return list(map(args)) File "/home/mint21/.local/lib/python3.10/site-packages/dali/dali_main.py", line 199, in _input_plugin_helper plugin_transactions = input_plugin.load() File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 157, in load process_account_result_list = pool.map(self._process_account, accounts) File "/usr/lib/python3.10/multiprocessing/pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/lib/python3.10/multiprocessing/pool.py", line 771, in get raise self._value File "/usr/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(args, kwds)) File "/usr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar return list(map(*args)) File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 185, in _process_account self._process_transfer(transaction, currency, intra_transaction_list) File "/home/mint21/.local/lib/python3.10/site-packages/dali/plugin/input/rest/coinbase_pro.py", line 224, in _process_transfer transfer_id: str = transaction_details[_TRANSFER_ID] KeyError: 'transfer_id'

[eprbell]

Thanks for the additional details. The other porfolio crashing makes sense (it's the same issue): clearly we are looking at a missing feature, more than a bug (transfers between CBPro "portfolios" are not supported yet). We may have to iterate a few times until we get all details right.

Which brings us to experiment 2. To run it:

• clone the DaLI source: git clone https://github.com/eprbell/dali-rp2
• setup the environment for development by following these instructions: https://github.com/eprbell/dali-rp2/blob/main/README.dev.md#setup
• save the diff from my previous message to a file, say /tmp/patch.diff
• cd to the newly cloned DaLI source directory and apply the diff: git apply /tmp/patch.diff
• run experiment 2

The goal of experiment 2 is to try to get past the key error and log the transfer data structure (see https://github.com/eprbell/dali-rp2/blob/main/src/dali/plugin/input/rest/coinbase_pro.py#L230).

[gbtorrance]

Ran into some problems with the environment setup.

First of all, I needed to modify these two lines since the "bin" folder for me is in ".venv/local" . .venv/local/bin/activate .venv/local/bin/pip3 install -e '.[dev]'

Then, when running this line, it seemed to be working, and then I eventually got the logs below. .venv/local/bin/pip3 install -e '.[dev]'

Not sure what's going on. BTW, I'm running within a Linux Mint VirtualBox guest (on a Linux Mint host). Unfortunately, my host OS is a few years old, and I was having all sorts of Python related problems (as it requires that both 2.7 and 3.x be installed). That's why I switched to using VirtualBox -- so I could install the latest Mint and not have to deal with out of date Python issues. I'm not sure if the issue below is caused by VirtualBox, but it could be. (I'm working in a "local" folder in the guest, not a shared one on the host.)

I'm happy to keep trying to get this resolved so I can help with the testing, but just so you know, I've decided I won't be able to use RP2 for this filing season. I literally have 11 exchanges/wallets, and RP2 only supports one of them (CoinbasePro). I must have been mad to think I could somehow get it to work for all the others before needing to file in mid-October. I've decided to use CoinLedger.io for now. Hoping it will meet my needs. RP2 is a great project, and I hope I can maybe use it in future, but reality dictates I just won't be able to this time.

  Attempting uninstall: dali-rp2
    Found existing installation: dali-rp2 0.5.2
    Uninstalling dali-rp2-0.5.2:
      Successfully uninstalled dali-rp2-0.5.2
  Running setup.py develop for dali-rp2
    error: subprocess-exited-with-error

    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [32 lines of output]
        running develop
        /usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        WARNING: The user site-packages directory is disabled.
        /usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        error: can't create or remove files in install directory

        The following error occurred while trying to add or remove files in the
        installation directory:

            [Errno 13] Permission denied: '/usr/local/lib/python3.10/dist-packages/test-easy-install-9093.write-test'

        The installation directory you specified (via --install-dir, --prefix, or
        the distutils default setting) was:

            /usr/local/lib/python3.10/dist-packages/

        Perhaps your account does not have write access to this directory?  If the
        installation directory is a system-owned directory, you may need to sign in
        as the administrator or "root" account.  If you do not have administrative
        access to this machine, you may wish to choose a different installation
        directory, preferably one that is listed in your PYTHONPATH environment
        variable.

        For information on other options, you may wish to consult the
        documentation at:

          https://setuptools.pypa.io/en/latest/deprecated/easy_install.html

        Please make the appropriate changes for your system and try again.

        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  WARNING: No metadata found in /home/mint21/.local/lib/python3.10/site-packages
  Rolling back uninstall of dali-rp2
  Moving to /home/mint21/.local/bin/dali_us
   from /tmp/pip-uninstall-fb46acjc/dali_us
  Moving to /home/mint21/.local/lib/python3.10/site-packages/dali/
   from /home/mint21/.local/lib/python3.10/site-packages/~ali
  Moving to /home/mint21/.local/lib/python3.10/site-packages/dali_rp2-0.5.2.dist-info/
   from /home/mint21/.local/lib/python3.10/site-packages/~ali_rp2-0.5.2.dist-info
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [32 lines of output]
    running develop
    /usr/lib/python3/dist-packages/setuptools/command/easy_install.py:158: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    WARNING: The user site-packages directory is disabled.
    /usr/lib/python3/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    error: can't create or remove files in install directory

    The following error occurred while trying to add or remove files in the
    installation directory:

        [Errno 13] Permission denied: '/usr/local/lib/python3.10/dist-packages/test-easy-install-9093.write-test'

    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:

        /usr/local/lib/python3.10/dist-packages/

    Perhaps your account does not have write access to this directory?  If the
    installation directory is a system-owned directory, you may need to sign in
    as the administrator or "root" account.  If you do not have administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.

    For information on other options, you may wish to consult the
    documentation at:

      https://setuptools.pypa.io/en/latest/deprecated/easy_install.html

    Please make the appropriate changes for your system and try again.

    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

[eprbell]

Looks like some peculiar behavior in Linux Mint: it's trying to write to /usr/local/lib for some reason, which fails (I assume you're not logged in as root). Did the activate step work OK? What happens if you type: which python?

As for you using a different software this time around, I completely understand: RP2 is a work in progress and it's meant to be a community effort (as new plugins or bits of logic are needed, people can fill them in by submitting PRs), so it'll get more and more complete over time. However you have a deadline that's fast approaching and you need to do your taxes ASAP: that's totally reasonable.

In any case, thanks for bringing up this missing bit of implementation in CBPro (I wasn't aware of it). If you are able to solve the Linux Mint setup issues and want to continue helping on this issue that's great, but if you need to move on to other things, no worries!

[gbtorrance]

Thank you for the reply, and sorry for the delay getting back to you. Life has been kind of crazy and stressful lately. As much as I'd like to try and get my Linux Mint issues resolved and help with testing for the feature implementation, I don't think I'm able to right now. (Maybe at some point.) Sorry.

Thank you for creating this product and for working with me to analyze this particular issue. I hope RP2 and Dali-RP2 continue to grow and thrive in the years ahead. Cheers!

[topcoderasdf]

I was able to replicate the same error using Coinbase Pro's sandbox environment. The transaction is basically a transfer between two different portfolios within a Coinbase Pro account.

Steps to replicate (costs no money btw as we are using sandbox).

1. Go to https://public.sandbox.pro.coinbase.com and login.
2. Add any asset (BTC, USD, USDC, etc) to the default portfolio
3. Create another portfolio (by clicking the create button right next to All Portfolios) and move assets from the default portfolio to the newly created portfolio.
4. Change __API_URL: str = "https://api.pro.coinbase.com/" to __API_URL: str = "https://api-public.sandbox.pro.coinbase.com/" within coinbase_pro.py and run

[eprbell]

Thanks for the repro! That's quite useful. Unfortunately I have very few cycles to dedicate to this issue at the moment. If anybody would like to work on this I'll review PRs.