Indexers speed-ups

[dr-orlovsky]

The current logic of indexers is implemented in a sub-optimal (from the performance perspective) fashion: many requests for txes are repeated, significantly increasing validation time for the large wallet histories.

Specifically, things which can be improved:

• caching all transactions returned by Electrum get_history and using them in resolving previous transaction inputs instead of always querying get_transaction method
• adding transactions returned as a previous inputs for the cache, which will prevent repeated calls retrieving the same transactions later (if a tx have multiple outputs it can be reported as previous transaction for each of the outputs)
• complete implementation of the update method which can leverage the knowledge of past wallet transaction to significantly reduce the number of requests to the indexer

[will-bitlight]

Dear Maxim,

The bitlight team has recently begun focusing on this issue. After considering your valuable suggestions, we've formulated some design ideas. (Please feel free to correct us if anything is amiss.) We intend to refine this issue according to these concepts and will provide a more detailed description of the specific modifications in the final PR.

• Cache Introduction:

  • We've added IndexerCache to various Indexer structures, utilizing LRU for cache management. This approach prevents unlimited growth while retaining the most frequently accessed information (primarily transaction data, UTXOs, etc.), ensuring efficient memory management.
  • The cache is constructed using data sourced from different Indexer APIs.

• Optimization of the create method:

  • WalletCache is primarily built using cached data, reducing network requests.
  • For cache misses, data is fetched from the Indexer's API and the cache is updated accordingly.

• Implementation of the update method:

  • We've encapsulated a transaction Stats logic based on different Indexer APIs. Before each execution, it compares the Stats of the cached information with those obtained from the Indexer's API. Only incremental updates are retrieved, which are then used to update both IndexerCache and WalletCache.

• Expected Optimizations:

  • Reduction in redundant network requests.
  • Accelerated data access through caching.
  • Incremental update strategy, processing only new transactions.

• API Compatibility:

  • We aim to implement functionality within your existing function signatures, maintaining the library's consistency.

  You can assign the issue to me and I will execute the implementation

[dr-orlovsky]

I agree that cache is the best way.