I'm in process of porting the wallet logic from the ./demo crate to ./node/wallet module and want to make sure the logic of tracking UTXOs is correct. The version in the demo is not well-specified and looks more like a patchwork and does not work reliably with respect to unconfirmed (mempool'ed) transactions.
Definitions
Tx: a ZkVM transaction
Output: location of the value locked in a contract of a transaction that can be spent by an input in the child tx.
UTXO: unspent tx output.
Confirmed: presence in the blockchain.
Unconfirmed: absence in the blockchain, but possible presence in the mempool.
Address: a key controlled by this wallet, derived from a single "account xpub".
Incoming: other party's payment towards an address controlled by this wallet.
Outgoing: tx or output created by this wallet.
Change: an output that sends the rest of the value back to the wallet.
Requirements
The wallet must provide a list of spendable utxos through its API that consist of:
confirmed incoming payments,
confirmed change outputs, and
unconfirmed change outputs,
excluding unconfirmed incoming payments, as the wallet has no control over the chance of those not being double-spent in the future.
When the wallet makes a payment:
it needs to store the unconfirmed tx, so it will attempt to re-submit it to mempool whenever it falls out of it,
remember the change output, so it can be spent in the next payment,
remember the annotation for both the change and destination output, so this data can be rendered in the UI.
When the wallet receives a payment:
a newly observed (confirmed or unconfirmed) tx is checked against the list of known addresses (stored or on-the-fly derived from account xpub) to detect whether this tx is relevant for this wallet,
confirmed outputs (change + incoming) that can be spent by the wallet are added as confirmed spendable utxos.
unconfirmed outputs and their transactions are treated separately:
outgoing transactions have their change outputs marked as spendable.
incoming transactions have their payment output counting towards unconfirmed balance, but not used for spending.
When the wallet adds unconfirmed tx to the mempool:
if it fails to be added, it can only be because it's expired or double-spent: then associated outputs are removed from the balance. Note: this can only happen with incoming txs, as outgoing ones are never intentionally double-spent by the wallet (and the maxtime on wallet's txs is set to infinity).
Wallet can expect that the notifications about txs, from blocks or from mempool are coming in correct topological order: meaning, children come after parents. Likewise, attempts to re-add unconfirmed txs to mempool must happen parent-first.
Since the incoming unconfirmed txs can be double-spent by the originator, those outputs must not be used for outgoing payments, nor should be stored alongside confirmed outputs, so we properly evict them when we evict the transaction.
Design sketch
Single SQLite database for the wallet state.
accounts table that stores xpubs.
addresses table that stores derived addresses and account_id, with expiration date - incoming txs are matched against this table to find if they belong to any account.
incoming_txs: list of unconfirmed incoming txs, to be continuously re-added to mempool, or removed if they become invalid.
outgoing_txs: list of unconfirmed outgoing txs. Same treatment as for incoming, except we know that we are not going to double-spend these txs and can expect them to be confirmed some time in the future (if they are stuck with low-fees, CPFP strategy will allow bumping the fee).
pending_utxos: list of unconfirmed outputs that form an unconfirmed (and untrusted) balance. Linked to its tx via incoming_tx_id in the incoming_txs table.
confirmed_utxos: list of confirmed outputs that can be spent by the wallet. Does not contain unconfirmed outputs.
annotated_outputs table for UI purposes - contains every known output, whether incoming, outgoing or change. Used to display txs in the UI only, not used for calculating balances and spendable utxo subset.
annotated_txs - same as outputs, but for raw txs.
API sketch
add an account with xpub
get spendable utxos to compute balance
get unconfirmed incoming utxos to compute full pending unconfirmed balance
get a list of annotated txs for a given time period, or simply paginated.
feed new txs from blockchain or mempool, and absorb+annotate the relevant ones in appropriate DB tables.
query unconfirmed txs in topological order to add to mempool
remove unconfirmed incoming txs (when they are expired / conflicting - failed to be (re)added to the mempool)
oleganza
commented
4 years ago
Problem
I'm in process of porting the wallet logic from the
./democrate to./node/walletmodule and want to make sure the logic of tracking UTXOs is correct. The version in the demo is not well-specified and looks more like a patchwork and does not work reliably with respect to unconfirmed (mempool'ed) transactions.Definitions
Tx: a ZkVM transaction
Output: location of the value locked in a contract of a transaction that can be spent by an input in the child tx.
UTXO: unspent tx output.
Confirmed: presence in the blockchain.
Unconfirmed: absence in the blockchain, but possible presence in the mempool.
Address: a key controlled by this wallet, derived from a single "account xpub".
Incoming: other party's payment towards an address controlled by this wallet.
Outgoing: tx or output created by this wallet.
Change: an output that sends the rest of the value back to the wallet.
Requirements
Design sketch
accountstable that stores xpubs.addressestable that stores derived addresses and account_id, with expiration date - incoming txs are matched against this table to find if they belong to any account.incoming_txs: list of unconfirmed incoming txs, to be continuously re-added to mempool, or removed if they become invalid.outgoing_txs: list of unconfirmed outgoing txs. Same treatment as for incoming, except we know that we are not going to double-spend these txs and can expect them to be confirmed some time in the future (if they are stuck with low-fees, CPFP strategy will allow bumping the fee).pending_utxos: list of unconfirmed outputs that form an unconfirmed (and untrusted) balance. Linked to its tx viaincoming_tx_idin theincoming_txstable.confirmed_utxos: list of confirmed outputs that can be spent by the wallet. Does not contain unconfirmed outputs.annotated_outputstable for UI purposes - contains every known output, whether incoming, outgoing or change. Used to display txs in the UI only, not used for calculating balances and spendable utxo subset.annotated_txs- same as outputs, but for raw txs.API sketch