UTXO index

[FreeTrade]

I wanted to submit a feature idea for consideration and discussion.

Currently --fastsync and --addrindex are incompatible options as --addrindex requires all historical transactions to be available. However a lighter version, say --addrutxoindex that indexes only utxos by address should be possible to be made compatible with --fastsync.

This would be useful for the app I'm making (a personal, uncensorable, desktop version of Member), and might be useful for a lite electron type server where it served just utxos rather than full historical trx data for clients.

Generally, I think we want to be moving away from the idea of default node being a full archival node and moving towards the idea that a node can be fastsynced and provides current rather than historical data. This should also help to focus emphasis on reducing utxo bloat rather than worrying about total historical chain size.

[cpacia]

Not sure I'll have the time to do this but you're welcome to submit a PR.

If you look at blockchain/indexers/addrindex.go you'll see an example of how it's done. You just need to satisfy the indexer interface:

type Indexer interface {
    // Key returns the key of the index as a byte slice.
    Key() []byte

    // Name returns the human-readable name of the index.
    Name() string

    // Create is invoked when the indexer manager determines the index needs
    // to be created for the first time.
    Create(dbTx database.Tx) error

    // Migrate is invoked after Init and allows each index the opportunity
    // to perform any migrations as necessary. This should be a noop if
    // there are no migrations to perform.
    Migrate(db database.DB, interrupt <-chan struct{}) error

    // Init is invoked when the index manager is first initializing the
    // index.  This differs from the Create method in that it is called on
    // every load, including the case the index was just created.
    Init() error

    // ConnectBlock is invoked when a new block has been connected to the
    // main chain. The set of output spent within a block is also passed in
    // so indexers can access the pevious output scripts input spent if
    // required.
    ConnectBlock(database.Tx, *bchutil.Block, []blockchain.SpentTxOut) error

    // DisconnectBlock is invoked when a block has been disconnected from
    // the main chain. The set of outputs scripts that were spent within
    // this block is also returned so indexers can clean up the prior index
    // state for this block
    DisconnectBlock(database.Tx, *bchutil.Block, []blockchain.SpentTxOut) error
}

The db should probably map key(derived from address) => outpoint. Then the outpoints could just be looked up in the utxo set. Should be easier than the addr index.

[FreeTrade]

Thanks. It is helpful to have a clear interface to implement. That does seem structured and straightforward enough. However, I haven't worked with the Go language before so I'm uncertain of my ability to implement it in a timely and rigorous way.

[zquestz]

The only problem I see with this interface is that it doesn't have a way to be given a utxo snapshot... when we fast sync we won't get ConnectBlock and DisconnectBlock messages, there were no blocks downloaded. So we would probably need to extend the interface to also allow sending an explicit set of utxos...

[cpacia]

Yeah I think you're right