Async wallet scanner

[j-berman]

Ready for review.

I've observed the following speed-ups benchmarking versus wallet2:

Setup	% faster than wallet2
Remote node, clearnet	50-60% faster
Remote node, tor	40-45% faster
Local node	25-35% faster

Benchmark results can vary widely based on setup (client/node internet speeds and machines). Source for the scanner used to benchmark.

How it works

Assume the user's wallet must start scanning blocks from height 5000.

1. The scanner begins by launching 10 RPC requests in parallel to fetch chunks of blocks as follows:

request 0: { start_height: 5000, max_block_count: 20 }
request 1: { start_height: 5020, max_block_count: 20 }
...
request 9: { start_height: 5180, max_block_count: 20 }

2. As soon as any single request completes, the wallet immediately parses that chunk.
   • This is all in parallel. For example, as soon as request 7 responds, the wallet immediately begins parsing that chunk in parallel to any other chunks it's already parsing.
3. If a chunk does not include a total of max_block_count blocks, and the chunk is not the tip of the chain, this means there was a "gap" in the chunk request. The scanner launches another parallel RPC request to fill in the gap.
   • This gap can occur because the server will not return a chunk of blocks greater than 100mb or 20,000 txs via the /getblocks.bin RPC endpoint.
   • The gap is from (req.start_height + res.blocks.size()) to (req.start_height + req.max_block_count).
4. As soon as the scanner finishes parsing the chunk, it immediately submits another parallel RPC request.
5. In parallel, the scanner identifies a user's received (and spent) enotes in each chunk.
   • For example, as soon as request 7 responds and the wallet parses it, the wallet scans that chunk in parallel to any other chunks it's already scanning.
6. Once a single chunk is fully scanned locally, the scanner launches a parallel task to fetch and scan the next chunk.
7. Once the scanner reaches the tip of the chain (the terminal chunk), the scanner terminates.

Some technical highlights

• The wallet scanner in this PR is backwards compatible with existing daemons (though it would need to point to a daemon running the updates in this PR to realize the perf speed-up).
• On error cases such as the daemon going offline, the same wallet errors that wallet2 uses (that the wallet API expects) are propagated up to the higher-level Seraphis lib.
• The implementation in this PR uses an http client connection pool (reusing the epee http client) to support parallel network requests (related).
• A developer using the scanner can "bring their own blocks/network implementation" to the scanner by providing a callback function of the following type as a param to the async scanner constructor: std::function<bool(const cryptonote::COMMAND_RPC_GET_BLOCKS_FAST::request, cryptonote::COMMAND_RPC_GET_BLOCKS_FAST::response)>.
  • Thanks to this approach, in relatively short order, I could test swapping the epee http client for libcurl's easy interface (souce).

Major TODO's for this PR

• Cleanup the formatting, code structure, code names and commit history.
  • DONE
• Implement backwards compatibility so the scanner will work pointing to nodes running today.
  • DONE: also implemented multithreaded scanning so perf won't drop if pointing to a node not running this PR.
• Add unit tests.
  • DONE:
    • Added functional tests that the async scanner can scan the chain and identify various legacy txs constructed by wallet2.
    • Added core tests that scanning util can parse all legacy enote versions from different legacy tx types.
      • It would be nice to test that the scanner can fully scan all legacy version enotes (and decode amounts).
• Try to smooth out the scanner's progress.
  • The scanner completes scanning many chunks in intermittent waves and then sits waiting (uncomfortably) until the next wave of chunks; it doesn't scan in a steady stream.
  • DONE: reduced the max_chunk_size_hint from 1000 to 20 to fetch chunks 20 blocks at a time. Overall time to scan remains the same (potentially slightly faster) and feedback to the user on scanner progress is significantly smoother.

Other TODO's for this PR

• Pre-RCT enotes can be scanned by the Seraphis wallet lib, however, the Seraphis wallet lib does not scan enough data to be able to spend them via legacy wallet2 today (without an RPC request that would reveal the enote to the daemon).
  • The Seraphis wallet lib would at minimum need to keep track of the global output ID by amount in order to capture enough data to pass to wallet2 to spend a pre-RCT enote before the Seraphis fork.
    • The Seraphis wallet lib would need to keep track of the global output ID by amount forever if pre-RCT outputs must be spent in separate rings from RCT outputs (related).
  • UPDATE: this is being worked on here https://github.com/UkoeHB/monero/pull/40
• Move "mock" implementations to the "production-ready" sections they belong.
  • DONE: moved production-ready implementations into expected sections. Mocks that are not ready for production are kept as mocks.
• Complete TODO's in the code.
  • DONE: TODO's necessary an initial PR are done.
• As soon as the scanner encounters an error or encounters the terminal chunk, it should immediately cancel any pending tasks it's waiting on and error out quickly.
  • DONE: small chunk requests (1000 blocks at a time -> 20 blocks at a time) took care of this, the async scanner cancels pending tasks and errors out quickly.
• Remove the blockchain scanner utility from this PR (and all changes to existing wallet functions for this utility). It's there to help see how the scanner works.
  • DONE: the functional test now demonstrates how the scanner works)

Misc.

There are a few things from this PR I can PR to the Monero repo today in bite-sized PR's:

• Moving wallet error logic out of wallet2 and into wallet/wallet_errors.h. I did this so I could reuse wallet2's error logic in this PR, which the higher level wallet API also expects.
• Modifying the get_blocks.bin daemon RPC endpoint:
  • I added an optional max_block_count param to the request (not a breaking change)
  • If the request includes a start_height that is greater than the chain tip and the request field fail_on_high_height is false, then instead of erroring, the server returns empty blocks and the chain height.
  • The response includes an optional top block hash field in addition to the chain height.
    • TODO: make this field optional in the response so it's not a breaking change. (DONE)
• A new functional test for direct wallet2 <> live RPC daemon interactions (and a simple daemon RPC client helper that the test uses).

[UkoeHB]

Nice to see this coming along :)

[j-berman]

Probably worth holding off on a deeper review until it's ready, almost there :)

[j-berman]

Ready for review :)

Noting there are 2 significant aspects of this scanner that will need to be dealt with. I think it'll be easier to handle in follow-up PR's.

1. The subaddress lookahead.

• The EnoteFindingContextMockLegacy takes as input a constant legacy_subaddress_map to see enotes received to subaddresses.
  • This means all subaddresses the async scanner will scan must be known to the wallet before it starts scanning.
• However, wallet2 currently implements a lookahead that expands when scanning tx outputs serially.
  • If wallet2 identifies a received output to a fresh subaddress, it then expands the lookahead to identify subsequent outputs received to higher indexed subaddresses.
  • This means the async scanner can miss txs to a subaddress that wallet2 would pick up.
• In order to match wallet2's behavior with max performance, a LegacyBasicEnoteRecord could instead be view tag matched enotes and their nominal_spendkey, rather than known-owned view scanned enotes.
  • As the scanner then consumes a chunk of LegacyBasicEnoteRecord's serially in process_chunk_full_legacy and identifies owned enotes, it then expands the suabddress lookahead as needed.

2. Pool handling.

• The async scanner does not deal with pool handling in this PR.
• The Seraphis lib scanner does a "scanning pass" in this order: 1) scan chain, 2) scan pool, 3) scan chain again in case the chain advanced.
• wallet2 does pool handling through the same single RPC request as fetching blocks from the daemon RPC (/getblocks.bin) in order to reduce round trips to the daemon.
• The async scanner design could change to optimally handle pool txs in line with the way wallet2 currently does it. Some scratchpad thoughts:
  • I think it would make sense to re-structure the Seraphis lib to simply do a single "scanning pass" (scan chain then pool).
  • The Seraphis lib currently assumes it's finished scanning upon encountering an empty chunk. The async scanner could pass an empty chunk back once all pool txs have been fetched.
  • Alternatively, the AsyncScanContextLegacy could cache pool txs and then that cache can be reused for get_nonledger_chunk.

[j-berman]

To reviewers: I suggest going commit-by-commit and working up to the final async scanner.

10/13 of the commits are theoretically independent and can be PR'd separately.

If the design generally looks solid, I can divide it up into smaller PR's to ease review. Additionally we can PR the daemon RPC changes to monerod at any time, so that daemons would be prepared for this scanner well in advance of releasing the Seraphis lib in a functional wallet to end-users.

[UkoeHB]

I think it would make sense to re-structure the Seraphis lib to simply do a single "scanning pass" (scan chain then pool).

I intentionally added a follow-up pass because there is a race condition between chain scanning and pool scanning that can cause mined blocks to contain txs invisible to a wallet even after it claims to have scanned the pool.

[j-berman]

Opened a separate issue to discuss since I don't think it needs to affect this PR: https://github.com/UkoeHB/monero/issues/41

[UkoeHB]

Once this is merged, please open as many PRs to monero core as possible to reduce the file overlap between this branch and core.

[j-berman]

Submitted PR's for the simple modifications to the core monero repo for the async scanner. If daemons run those changes, they'll be fully compatible with the async scanner in this PR.