Open rajarshimaitra opened 2 years ago
Note that the main goal for this task is not really to add more descriptors to the Wallet
structure, but mainly to find a way to group multiple wallets into a "super wallet" that can operate on all of them at the same time, for instance by creating a transaction that spends utxos from all of them
Noted. Paraphrased the above in the description. Let me know if it can be elaborated more..
Note that the main goal for this task is not really to add more descriptors to the
Wallet
structure, but mainly to find a way to group multiple wallets into a "super wallet" that can operate on all of them at the same time, for instance by creating a transaction that spends utxos from all of them
I think the simplest solution is create a API that allows you to create a PSBT using the databases of multiple wallets. Could even be a TxBuilder
method add_utxos_from_wallet(wallet)
. Internally this could just take list_utxos
do a weak version of add_foreign_utxo
where the UTXO would be a non-mandatory spend. This would give you all the functionality that isn't available in BDK at the moment.
Of course this leaves you to manually add the balances of the individual wallets to find the joint balance etc. But more sophisticated things can be done later. In my mind wallets already have two descriptors with different semantics. I don't see why they couldn't have three or four or twenty four all in a Vec.
I think once we converge on an approach, the description can be elaborated further. I feel this will be a little difficult project for the students and they would need as much context as possible to tackle this issue.
Thanks for all the inputs @LLFourn @afilini , I will keep following this thread and update the project template accordingly.
In my mind wallets already have two descriptors with different semantics. I don't see why they couldn't have three or four or twenty four all in a Vec.
Yes, what worries me is how complex it can get to define all the different semantics for n descriptors inside a wallet. If this "super-wallet" thing turns out well we could also consider changing the wallets to only have one descriptor (which would simplify a ton of stuff) and then handling the semantics outside (for instance the super-wallet create_tx
knows that it needs to generate the change address from one specific sub-wallet that is designated as the "internal" one).
I would be interested in having a go at this issue.
Per our discussion from bitcoindevkit/.github#51 the new bdk
module now has a Wallet
using the new KeychainTracker
which can track keychains for multiple descriptors; it currently only uses two: External
and optional Internal
. My initial thought was to keep this as is to not complicate the bdk::Wallet
API. But with prompting from @rajarshimaitra I've taken a deeper look and think supporting multiple descriptors in Wallet
can simplify the wallet internals and improve the user facing API. The biggest down side I see is this is a fair bit of work, but if we're going to make major API changes now is the time.
Below is my high-level take on what would need to be done, what am I missing? Any reasons not to make this change?
Becomes a collection of descriptor signers with corresponding keychain_tracker, persist, network, secp. Provides typical wallet functions across multiple descriptors for getting addresses, balances, listing transactions and utxos, creating new transactions via TxBuilder
, and signing and finalizing PSBTs.
K
type param with traits needed by KeychainTracker
's K typechange_signers
field, only have signers
new
to take a Vec
of tuple (K
, E: IntoWalletDescriptor
)K
param to specify which keychain to use
see #898Vec
of K
param for filtered resultsKeychainKind.External
and KeychainKind.Internal
descriptor Wallet
Wallet
Create spending and fee bump transactions for a multi descriptor Wallet
.
Vec
of K
keychains allowed to spend UTXOs from, with optional policy pathsVec
of K
keychains not allowed to spend UTXOs fromK
keychain to use for change outputchange_policy
, it can be represented with abovePer my chat with @evanlinjin I've done some more digging into how the Core wallet works, in particular regarding privacy and multiple descriptors, see https://github.com/bitcoindevkit/bdk/issues/918#issuecomment-1491221046. I also want to incorporate @rajarshimaitra suggestion for keeping simplified wallet functions for users who only have external + (optional) internal descriptors. My original ideas above will need to be updated.
From the above research plus goal of making Wallet
easier to use for the basic two descriptor wallet scenario I propose a few modifications to above:
DefaultKeychainKind
trait requirement on K
to set a keychain kind (External or Internal) and is_default bool
change_policy
since it would still be needed to decide which kind of keychains to get utxos fromThe basic example should be setup similar to what Core does for a new single key descriptor wallet:
pkh()
, sh(wpkh())
, tr()
and wpkh()
An advanced example would be similar to above but also include using non-default descriptors. A scenario I'm thinking of is migrating utxos from an old wallet to a new wallet. We can do this by adding the old wallet descriptors as non-default descriptors to our new wallet. Over time old wallet utxos will be spent, but wallet continues to monitor old chains for any received utxos to old descriptors and spend them when needed with change back to new descriptors:
EDIT: changed above examples from 3+3 to 4+4 descriptors per @vladimirfomene comment below.
From the above research plus goal of making
Wallet
easier to use for the basic two descriptor wallet scenario I propose a few modifications to above:Wallet
1. add a `DefaultKeychainKind` trait requirement on `K` to set a keychain kind (External or Internal) and is_default `bool`
What's wrong with the usual Default
trait here?
add a DefaultKeychainKind trait requirement on K to set a keychain kind (External or Internal) and is_default bool
I was thinking of
struct Wallet<D = (), K = KeychainKind>
, In that way we can keep all the existing APIs in
impl <D : PeristentBackend> Wallet<D> {}
And all the new multi descriptor API in
impl<D: PersistentBackend, K: Debug + Clone + Ord> Wallet <D, K> {}
scope.
In this way all the users who don't care about keychains and used bdk in previous way can use the Wallet::new()
constructor without specifying anything related to keychains
and the current implementation of new will take care of using the KeychainKind
enum internally to manage the two keychains.
I think @vladimirfomene tried this pattern out and and it seemed to work. Vlad can you confirm on this?
1. add a `DefaultKeychainKind` trait requirement on `K` to set a keychain kind (External or Internal) and is_default `bool`
What's wrong with the usual
Default
trait here?
I'm not thinking of a Default
as in the default variant of K
. A better name for the trait I have in mind is ActiveKeychainKind
to filter K
for ones that are "active" and "external" or "internal" when I need to pick a 'K' for new addresses.
If we don't want to impose any additional traits on K
, the other way I can think to do it is for a non-default K
, multi-descriptor Wallet
(as @rajarshimaitra suggests above) we add a K
param to the functions that need it, like getting a new receive address, or for change address in the TxBuilder.
Can we just make the user be explicit about which addresses they want rather than it being implicitly defined by a trait? It seems to me like the application should always know when requesting one. When creating a tx we could force the user to tell us about which keychain they want to use to derive a change address too.
add a DefaultKeychainKind trait requirement on K to set a keychain kind (External or Internal) and is_default bool
I was thinking of
struct Wallet<D = (), K = KeychainKind>
, In that way we can keep all the existing APIs inimpl <D : PeristentBackend> Wallet<D> {}
And all the new multi descriptor API in
impl<D: PersistentBackend, K: Debug + Clone + Ord> Wallet <D, K> {}
scope.In this way all the users who don't care about keychains and used bdk in previous way can use the
Wallet::new()
constructor without specifying anything related tokeychains
and the current implementation of new will take care of using theKeychainKind
enum internally to manage the two keychains.I think @vladimirfomene tried this pattern out and and it seemed to work. Vlad can you confirm on this?
I find this strategy workable. Let me sync with Steve to better understand how he thinks about this.
Can we just make the user be explicit about which addresses they want rather than it being implicitly defined by a trait? It seems to me like the application should always know when requesting one. When creating a tx we could force the user to tell us about which keychain they want to use to derive a change address too.
If I understand correctly, that's exactly the plan. Force the user to specify a keychain to get addresses and create transactions from them. And this should not require any extra traits. We just need to manage the generics right while implementing multi-desc APIs, and provide a default for users who don't care about keychains/multi-descs.
Maybe eventually, we should not have the "default" wallet behavior and treat everything as multi-desc and always force users to specify keychains. But it will be a smoother transition to keep the default alive for now, as that makes BDK wallet really easy to spawn as a personal wallet setup.
I'm not thinking of a Default as in the default variant of K. A better name for the trait I have in mind is ActiveKeychainKind to filter K for ones that are "active" and "external" or "internal" when I need to pick a 'K' for new addresses.
I think this will be a pretty massive break for the API. The concept of keychain wasn't present before.
I think
We can add the new APIs where users are forced to specify keychains, and can add their custom keychains of any kind that they like.
Vlad is close to coming up with the concept code, (waiting on my review). At this point, it would be helpful to have the code to talk over.
Examples
The basic example should be setup similar to what Core does for a new single key descriptor wallet:
- 3 external default single key descriptors, one for each script type
- 3 internal default single descriptors, one for each script type
- get new deposit addresses from default external descriptor for requested script type
- get balance, list transactions and utxos for all wallet descriptors
- use appropriate internal descriptor for change, depending on the script type of the transaction output
@notmandatory , I think you meant to say 4 external and 4 internal default. The core wallet has 4 script types for its DescriptorScriptPubKeyMan
. 3 descriptors is for the LegacyScriptPubKeyMan
.
OK, I agree with above that we don't need any new trait on K
, and should go with struct Wallet<D = (), K = KeychainKind>
and impl <D : PeristentBackend> Wallet<D> {}
with functions that let the users not worry about multiple descriptors. Then for impl<D: PersistentBackend, K: Debug + Clone + Ord> Wallet <D, K> {}
have equivalent functions for advanced users where they will need to specify which K
for some functions.
@notmandatory , I think you meant to say 4 external and 4 internal default. The core wallet has 4 script types for its DescriptorScriptPubKeyMan. 3 descriptors is for the LegacyScriptPubKeyMan.
@vladimirfomene ah yes, you are correct, I must have been looking at an old pre-TR PR. I did a test just now to confirm and for a new default descriptor wallet Core creates 8 descriptors, internal + external for: pkh()
, sh(wpkh())
, tr()
and wpkh()
. I'll edit my above comment example.
With @rajarshimaitra and @notmandatory, we have settled on an initial approach to implementing a multi-descriptor wallet in BDK. Here are the highlights:
K
(K
has to be Ord + Clone + Debug as the generic keychain identifier in TxoutIndex) and give it a default type of KeychainKind
so that we can have a default wallet that just supports two descriptors (internal and external). Vec<K, E: IntoWalletDescriptor>
in its constructor and initialize the wallet.
Description
In order for BDK to extend beyond just a Bitcoin wallet, it needs a more powerful descriptor capability. This project aims at improving the descriptors used in the library. Currently BDK uses only a single descriptor for a single wallet. Giving a new descriptor to BDK will create a brand new wallet. The goal of this project is to create a wallet that can handle multiple descriptors within itself (a "super wallet"), and can send funds from any or all of them together. **Expected Outcomes** - Through understanding of [descriptors](https://github.com/bitcoin/bitcoin/blob/master/doc/descriptors.md). - Basic understanding of the [rust-miniscript](https://github.com/rust-bitcoin/rust-miniscript) library. - Multi descriptor capability in BDK. **Resources** **Skills Required**Mentor(s)
Difficulty: Hard
Competency Test (optional)