search

eprbell / dali-rp2

DaLI (Data Loader Interface) is a data loader and input generator for RP2 (https://pypi.org/project/rp2), the privacy-focused, free, open-source cryptocurrency tax calculator: DaLI removes the need to manually prepare RP2 input files. Just like RP2, DaLI is also free, open-source and it prioritizes user privacy.
https://pypi.org/project/dali-rp2/
Apache License 2.0
64 stars 42 forks source link

Fatal exception: No path found from AR to USD #133

Open kiates opened 1 year ago

kiates commented 1 year ago

Received No path found for AR to USD. Please open an issue at https://github.com/eprbell/dali-rp2/issues. when processing input from Pionex using the csv.pionex plug-in:

2023-04-01 14:13:18,834/dali/DEBUG: Fetched high conversion rate 1.0005 for 2022-01-13 15:22:14+00:00/USDT->USD from cache of plugin Historic-Crypto: HistoricalBar(duration=datetime.timedelta(seconds=60), timestamp=Timestamp('2022-01-13 15:22:00+0000', tz='UTC'), open=Decimal('1.0004'), high=Decimal('1.0005'), low=Decimal('1.0004'), close=Decimal('1.0005'), volume=Decimal('31751.38')) 2023-04-01 14:13:18,834/dali/DEBUG: Unresolvable transaction (no unique_id): InTransaction: plugin=Pionex_CSV unique_id=__unknown raw_data=01/13/2022 15:22:14,0.25261294,USDT,0.00490807,AR,0,USDT, timestamp=2022-01-13 15:22:14+0000 asset=USDT exchange=Pionex holder=Chad Yates transaction_type=Buy spot_price=1.0005 crypto_in=0.25261294 fiat_fee=None fiat_in_no_fee=None fiat_in_with_fee=None notes=high spot_price read from Historic-Crypto plugin; 2023-04-01 14:13:19,325/CCXT-converter/high/DEBUG: Converting AR to USD 2023-04-01 14:13:19,325/CCXT-converter/high/DEBUG: Using default exchange Kraken type for Pionex. 2023-04-01 14:13:19,325/CCXT-converter/high/DEBUG: No path found for AR to USD. Please open an issue at https://github.com/eprbell/dali-rp2/issues. 2023-04-01 14:13:19,325/dali/ERROR: Fatal exception occurred: Traceback (most recent call last): File "C:\Users\cyates\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\dali\dali_main.py", line 169, in _dali_main_internal resolved_transactions: List[AbstractTransaction] = resolve_transactions(transactions, dali_configuration, args.read_spot_price_from_web) File "C:\Users\cyates\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\dali\transaction_resolver.py", line 246, in resolve_transactions transaction = _update_spot_price_from_web(transaction, global_configuration) File "C:\Users\cyates\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\dali\transaction_resolver.py", line 144, in _update_spot_price_from_web raise RP2RuntimeError( rp2.rp2_error.RP2RuntimeError: Spot price for __unknown:2022-01-13 15:22:14+00:00:AR->USD not found on any pair converter plugin 2023-04-01 14:13:19,328/dali/INFO: Log file: ./log/rp2_2023_04_01_14_02_25_617960.log 2023-04-01 14:13:19,328/dali/INFO: Generated output directory: dali-output 2023-04-01 14:13:19,328/dali/INFO: Done

kiates commented 1 year ago

On another run (the same file), I got this one instead for a different coin...

No path found for MBTC to USD. Please open an issue at https://github.com/eprbell/dali-rp2/issues.

Not sure why it's non-deterministic. Does the csv importer use multiple threads by default?

eprbell commented 1 year ago

Looks like you're using the default pair converter (Historic-Crypto), which doesn't have all markets. You'll want to try @macanudo527's CCXT-based pair converters, which implement a fancy price routing algorithm for cases where the pair is not directly available.

macanudo527 commented 1 year ago

I think he is using the CCXT converter but MBTC and AR aren't listed on Kraken the default source for pricing. I'll have to add alternative exchanges for them. I'll try to get to it soon.

kiates commented 1 year ago

AR -> USD

I was able to work around the AR -> USD case by just specifying for it to use Gate as the default ccxt exchange:

[dali.plugin.pair_converter.ccxt] historical_price_type = nearest default_exchange = Gate

I tried just adjusting the nested dictionaries that seem to be designed to allow overriding which exchange to fetch the spot price from. So I added a "AR": "USD" key/value to the 1st layer, and a "ARUSD": _GATE to the 2nd layer, but that didn't seem to work. Don't know why.

MBTC -> USD

The MBTC -> USD case is more difficult as it appears to be a Pionex-only thing. An MBTC is 1/1000th of a BTC by definition, and is probably something they made internally. I remember making BTC grid bots there and they would suggest switching behind the scense to MBTC to "improve fund utilization or something".

For the moment, I'm attempting to hack around it by just watching for MBTC/USD from/to pairs and then fetching the BTC/USD price instead and then adjusting the result by dividing by 1000.

Going a bit further, one could envision a set of rules that are just based on expressions. So for example, given a from asset of MBTC, the software could automatically know that it is 1/1000 of a BTC and register that conversion factor for MBTC -> BTC in the graph system -- which I don't fully understand yet, and it doesn't help that I'm not really a python programmer. Then in theory the current design would be able to walk from MBTC -> BTC -> USD or whatever else it needed to get to.

kiates commented 1 year ago

Ran into another that it couldn't resolve: NEO->USD (using Kraken instead of Gate)

eprbell commented 1 year ago

AR -> USD

I was able to work around the AR -> USD case by just specifying for it to use Gate as the default ccxt exchange:

[dali.plugin.pair_converter.ccxt] historical_price_type = nearest default_exchange = Gate

I tried just adjusting the nested dictionaries that seem to be designed to allow overriding which exchange to fetch the spot price from. So I added a "AR": "USD" key/value to the 1st layer, and a "ARUSD": _GATE to the 2nd layer, but that didn't seem to work. Don't know why.

MBTC -> USD

The MBTC -> USD case is more difficult as it appears to be a Pionex-only thing. An MBTC is 1/1000th of a BTC by definition, and is probably something they made internally. I remember making BTC grid bots there and they would suggest switching behind the scense to MBTC to "improve fund utilization or something".

For the moment, I'm attempting to hack around it by just watching for MBTC/USD from/to pairs and then fetching the BTC/USD price instead and then adjusting the result by dividing by 1000.

Going a bit further, one could envision a set of rules that are just based on expressions. So for example, given a from asset of MBTC, the software could automatically know that it is 1/1000 of a BTC and register that conversion factor for MBTC -> BTC in the graph system -- which I don't fully understand yet, and it doesn't help that I'm not really a python programmer. Then in theory the current design would be able to walk from MBTC -> BTC -> USD or whatever else it needed to get to.

That's an interesting case. Basically it would be an exchange-specific map of "artificial assets": "artificial asset name" -> ("real asset", transformation())

So on Pionex, given an artificial asset like MBTC the map would point to a tuple (BTC, divide_by_1000()). This is just to point out the functionality, but the implementation would use an ArtificialAsset class, with a get_real_asset() and transform_value() methods. Note that this mapping would only be defined in the context of Pionex (it's not global). So if in the future an actual new asset called MBTC appears on the market, the above mapping would not be applied outside of Pionex.

kiates commented 1 year ago

And here is another pair that couldn't resolve: BETA->USD

At this point I'm using the the following in the pair converter portion:

[dali.plugin.pair_converter.ccxt] historical_price_type = nearest default_exchange = Gate

[dali.plugin.pair_converter.historic_crypto] historical_price_type = nearest

It's my understanding that you can have two as long as they aren't the same type.

I'm wondering if there is a way to make it put some kind of Keyword.UNKNOWN state instead of hard failing. Then at least it could finish processing, and leave you with a list of things that you may have to manually figure out and adjust.

eprbell commented 1 year ago

@macanudo527, what is the process of adding new routes? I'm wondering if we can make it easy for people to add their own routes when this kind of problem happens or if we can document the process of adding routes.

@kiates, you can probably hack your local tree to output UNKNOWN by modifying this small function: https://github.com/eprbell/dali-rp2/blob/main/src/dali/transaction_resolver.py#L171

I'm not sure we'll want to merge that into the code though: the idea is that DaLI tries hard to generate correct files or it fails. This reduces support effort for the dev team.

eprbell commented 1 year ago

Also, yes, you can have multiple pair converters and they will be checked in order until a price is found (or until the list of pair converters is exhausted).

macanudo527 commented 1 year ago

@macanudo527, what is the process of adding new routes? I'm wondering if we can make it easy for people to add their own routes when this kind of problem happens or if we can document the process of adding routes.

Right now it is ad-hoc addition of markets, but there will have to be an automatic process of doing this since the lists will quickly become long and unwieldy. Right now the priority is replacing the graph system with djikstra and a few more optimizations like lazy load. Once it is cleaned up we can work on automatic discovery of markets.

@kiates the reason the AR-> USD market is failing is because it doesn't exist. You can add AR -> USDT though. Or, if you are in Japan, you can just make Binance your default and it will resolve a lot of these issues, I think. The market with the 2nd highest volume is Kucoin so you should add that one in if you can't use Binance. Be sure to check CoinMarketCap to see which exchange has the best market.

macanudo527 commented 1 year ago

That's an interesting case. Basically it would be an exchange-specific map of "artificial assets": "artificial asset name" -> ("real asset", transformation())

So on Pionex, given an artificial asset like MBTC the map would point to a tuple (BTC, divide_by_1000()). This is just to point out the functionality, but the implementation would use an ArtificialAsset class, with a get_real_asset() and transform_value() methods. Note that this mapping would only be defined in the context of Pionex (it's not global). So if in the future an actual new asset called MBTC appears on the market, the above mapping would not be applied outside of Pionex.

I don't think it needs to be that complicated actually. You can just add aliases with set prices to the graph. So, you add the market MBTC -> BTC locked at the spot price of 0.00000001 for pionex. This same mechanism can be used to resolve the issues with Kraken's X-assets (xbtc, xeth, etc...) By adding a xbtc -> BTC market and a locked spot price of 1. As well as renamed assets.

eprbell commented 1 year ago

That's an interesting case. Basically it would be an exchange-specific map of "artificial assets": "artificial asset name" -> ("real asset", transformation()) So on Pionex, given an artificial asset like MBTC the map would point to a tuple (BTC, divide_by_1000()). This is just to point out the functionality, but the implementation would use an ArtificialAsset class, with a get_real_asset() and transform_value() methods. Note that this mapping would only be defined in the context of Pionex (it's not global). So if in the future an actual new asset called MBTC appears on the market, the above mapping would not be applied outside of Pionex.

I don't think it needs to be that complicated actually. You can just add aliases with set prices to the graph. So, you add the market MBTC -> BTC locked at the spot price of 0.00000001 for pionex. This same mechanism can be used to resolve the issues with Kraken's X-assets (xbtc, xeth, etc...) By adding a xbtc -> BTC market and a locked spot price of 1. As well as renamed assets.

Nice: simpler and more elegant.

macanudo527 commented 1 year ago

@kiates It appears you are doing some of your trading with Binance.com. If that is the case, you should just set the default exchange to Binance. That would resolve most of the issues except mBTC. BETA appears to be only available on the Binance ecosystem, so it is hard to get a reliable price without using Binance.com

There shouldn't be issues using Binance.com for pricing unless you are in the US.

I'll work on adding AR for now, since it is a pretty popular asset, but I can't really add BETA without using Binance.com

kiates commented 1 year ago

I can't use binance as I'm USA. When I've tried their API rejects any requests as they are geo-locked. I might be able to use a VPN temporarily to get around it.

macanudo527 commented 1 year ago

I can't use binance as I'm USA. When I've tried their API rejects any requests as they are geo-locked. I might be able to use a VPN temporarily to get around it.

I wouldn't risk it. It seems like US authorities have made it their mission to seal off Binance.com in every way possible. And there is that new bill that is trying to make it illegal to access off-limits websites through VPNs.

The CSV files are still available for US citizens at least for a little while, so a CSV download plugin would help. It's on my to-do list, but not anytime soon. In the meantime, I don't think there is a way to accurately price BETA.

There was talk of implementing a Coingecko API pair converter that could be used in a last-resort situation like this. That is one way you could resolve it.

kiates commented 1 year ago

I do have access to Binance.US but I expect the coin list is very limited and not useful for this situation, plus Binance.US isn't supported yet.