DaLI (Data Loader Interface) is a data loader and input generator for RP2 (https://pypi.org/project/rp2), the privacy-focused, free, open-source cryptocurrency tax calculator: DaLI removes the need to manually prepare RP2 input files. Just like RP2, DaLI is also free, open-source and it prioritizes user privacy.
If we actually queried a source, store it in the cache so a future iteration can exit one step sooner
The issue with the above is that both plugins have the cache_key() of CCXT-converter and the following happens instead:
binance cache is saved to .dali_cache/CCXT-converter
kraken cache is saved to .dali_cache/CCXT-converter, over-writing the binance result
On next dali run, .dali_cache/CCXT-converter is loaded as the cache for both plugins
As exchange is part of the query key, the binance plugin cache is populated with kraken results and all queries against it fail. However, all queries against binance have to be repeated and the cache is effectively disabled for all but the lowest-priority ccxt plugin
Due to rate limiting at the source and the cache priority inversion, the above issue can significantly slow down dali on repeated runs and lead to almost no caching speedup.
A subtler issue can actually occur when setting fiat_priority if it is modified between dali runs: as the fiat exchange rate is applied before adding the result to the cache, any changes to fiat_priority will not be reflected.
The solution is to introduce a unique cache key that incorporates settings that may invalidate the cache and to require that users not have two identical pair converter plugins.
With this change, the ccxt_binance plugin would write to .dali_cache/CCXT-converter_Binance.com and ccxt_kraken plugin would write to .dali_cache/CCXT-converter_Kraken if fiat_priority is not set.
I'm indifferent on the format of the cache_key and am open to suggestions on better ones.
Assume a user has a config using multiple
ccxt
pair converter plugins - this can be done to express an explicit fallback order:The expected behavior in this case is:
dali
checks for cached result frombinance
binance
kraken
resultkraken
The issue with the above is that both plugins have the
cache_key()
ofCCXT-converter
and the following happens instead:binance
cache is saved to.dali_cache/CCXT-converter
kraken
cache is saved to.dali_cache/CCXT-converter
, over-writing thebinance
resultdali
run,.dali_cache/CCXT-converter
is loaded as the cache for both pluginsexchange
is part of the query key, thebinance
plugin cache is populated withkraken
results and all queries against it fail. However, all queries againstbinance
have to be repeated and the cache is effectively disabled for all but the lowest-priorityccxt
pluginDue to rate limiting at the source and the cache priority inversion, the above issue can significantly slow down
dali
on repeated runs and lead to almost no caching speedup.A subtler issue can actually occur when setting
fiat_priority
if it is modified betweendali
runs: as the fiat exchange rate is applied before adding the result to the cache, any changes tofiat_priority
will not be reflected.The solution is to introduce a unique cache key that incorporates settings that may invalidate the cache and to require that users not have two identical pair converter plugins.
With this change, the
ccxt_binance
plugin would write to.dali_cache/CCXT-converter_Binance.com
andccxt_kraken
plugin would write to.dali_cache/CCXT-converter_Kraken
iffiat_priority
is not set.I'm indifferent on the format of the
cache_key
and am open to suggestions on better ones.