Sparrow having difficalties syncing and maintaining Cormorant wallet with multiple Sparrow wallets

[eugenesan]

I might be just confused by how wallets work with Cormorant so please bear with me.

I've noticed that Cormorant waits for previous wallet sync to complete before start syncing newly loaded Sparrow wallets. (At least that's how it looks like visually and in the logs).

It would useful to be able to load multiple sparrow wallets in parallel.

If it's not possible to add new addresses/xpubs while single cormorant wallet is syncing maybe we should consider using multiple Core wallets for each Sparrow wallet since Bitcoin Core is capable of doing so with built in wallets.

In addition there seems be a problem with how Cormorant handles multiple wallets:

Test 1

1. Load wallet in Sparrow (Might be related, the wallet was previously synced with Fulcrum)
2. Wait for Core start syncing changes in cormorant
3. Shutdown Core and close Sparrow
4. Start core and open Sparrow
5. Open the wallet and observe Core and Sparrow treat the wallet as already synced

Test 2

1. Load wallet in Sparrow (Might be related, the wallet was previously synced with Fulcrum)
2. Wait for Core start syncing changes in cormorant
3. Load new wallet in Sparrow
4. Wait for wallet sync to finish
5. Observe the second wallet missing transactions

Test 3

1. In sparrow with multiple loaded wallets, force any wallet refresh and wait for the sync to finish
2. Close Sparrow and Core
3. Put aside existing cormorant wallet
4. Open Sparrow and load the same wallets
5. Open Core and wait for the new cormorant wallet to be created and synced
6. Compare previous cormorant wallet and the new one: some addresses, descriptors and transactions are missing from the cormorant wallet

Wallet refresh fixes the issue but requires another full sync. Is there a way to stop current sync when a new one is required?

P.S. Not directly related but; With pruned nodes, do we need to run Sparrow periodically, before Core prunes the blocks Sparrow missed? Or, Cormorant wallet in Core contains all the xpubs and Core will save new transaction for Sparrow fetch later?

[craigraw]

It would useful to be able to load multiple sparrow wallets in parallel.

My suggestion would be to use a Electrum server like Fulcrum for this use case. If you are opening and closing many wallets (new and old), this is likely to be a much better experience.

If it's not possible to add new addresses/xpubs while single cormorant wallet is syncing maybe we should consider using multiple Core wallets for each Sparrow wallet since Bitcoin Core is capable of doing so with built in wallets.

Unfortunately, this wouldn't help - see https://github.com/bitcoin/bitcoin/issues/11756

Test 1

Unfortunately there is no way for Sparrow to tell AFAIK if Core has been restarted during a wallet scan. The likely RPC call getwalletinfo does not return any information to indicate this - after the restart it simply indicates "scanning": false for the wallet. Rescans can be triggered in Sparrow by setting the birthday of the wallet to a little earlier, see Settings tab > Advanced.

Test 2

I'll need more specific instructions to recreate. FWIW I switch wallets between Core and Electrum frequently when testing, and I haven't observed this.

Test 3

Again, more specific information is needed to recreate.

Is there a way to stop current sync when a new one is required?

Not currently, although Core does offer the abortrescan RPC call. Implementing this in Sparrow might cause more issues than it solves though.

With pruned nodes, do we need to run Sparrow periodically, before Core prunes the blocks Sparrow missed?

No - so long as Core is kept running no transactions will be missed.

[eugenesan]

@craigraw Thank you for your detailed response.

I am not trying to frequently load multiple different wallets. All I need is a reliable procedure to switch to Cormorant, "sync-up" a few wallets and optionally perform Core pruning.

I understand that Full-Node/Fulcrum is the proper solution and that's what I am using ATM. But since not everyone can afford Full-Node and Sparrow documentation mentions support of pruned nodes with Cormorant, some users (including me) might want to use it.

I tried a few times and every time resulting Cormorant wallet in Core contained different set of addresses and descriptors. Seems like when multiple wallets are loaded, Cormorant mis-configures some descriptors or Core can't handle changes to cormorant wallet properly once rescanblockchain is ongoing. For example when querying several synced cormorant wallets using listreceivedbyaddress and listdescriptors:

1. Sometimes descriptors for the same xpub contain "active": false and sometimes "active": true
2. Sometimes some descriptors contain "internal": false
3. Every test cormorant wallet had different number of addresses. Both returned by listreceivedbyaddress and as descriptors "desc": "addr()
4. Each xpub descriptor appears twice, once with "next": 0, "next_index": 0 and once with "next": 1, "next_index": 1 Also, all wallets were missing some transactions when displayed in the core UI even after manual rescanblockchain.

Something must be done about this issue or at least there should be a warning regarding multi-wallet use with Cormorant.

[craigraw]

I believe you misinterpret the output you are seeing. For example:

Sometimes descriptors for the same xpub contain "active": false and sometimes "active": true Sometimes some descriptors contain "internal": false

Both of these flags are used by the Core wallet internally, and have no real impact on Cormorant. See below for why the "same xpub" appears.

Every test cormorant wallet had different number of addresses.

It's hard to comment on this without specific instructions to recreate it.

Each xpub descriptor appears twice, once with "next": 0, "next_index": 0 and once with "next": 1, "next_index": 1

Because multipath descriptor support is not yet merged, Cormorant imports 2 descriptors for each wallet - the receive chain, and the change chain. They look similar, but they are not the same.

Something must be done about this issue or at least there should be a warning regarding multi-wallet use with Cormorant.

You have yet to demonstrate any issue. The way to go about it is this - from a clean start (ie an empty cormorant wallet) perform and document a specific set of steps. Document the output and note any issues. Then start over and repeat the process, following exactly the same set of steps. Make sure you recreate the exact same output, or document how the output has changed. Use testnet to avoid using real funds, and share all the steps needed to recreate here.

[eugenesan]

Once again, thank you for detailed response.

I followed your advice and created 2 wallets on testnet4.

Here are the steps I took:

1. Create wallet1 in Core
2. Received 2 transactions from a faucet to wallet1 in Core (each transaction with unique address)
3. Created wallet2 in Sparrow (BIP39)
4. Imported wallet1 to Sparrow using BIP32 TPRIV
5. Stopped Core and closed Sparrow
6. Deleted all wallets from Core
7. Started Core and Sparrow
8. Waited until Sparrow re-created and synced cormorant wallet (was instant on testnet4)
9. Initiated transaction from wallet1 in Sparrow to wallet2 in Sparrow.
10. Waited until transaction was confirmed

Result:

1. cormorant wallet has 4 descriptors (2 for each TPUB) and 2 addresses via listdescriptors
2. cormorant wallet has 2 addresses via listreceivedbyaddress. Both addresses are from wallet1
3. cormorant wallet shows no sign of address from wallet2 that received confirmed transaction
4. Sparrow shows all 3 transactions (2 from faucet and 1 between wallets) and all 3 addresses

What's missing is the address in cormorant wallet that received transaction in step 9. Core also doesn't show the transaction from step 9. While sparrow shows everything as expected, I am concerned that if I would recreate from backup the 2 wallets in Sparrow, it wouldn't fetch all transactions using Cormorant library, especially if Core is pruned.

I guess the Core is not deriving the address despite having the corresponding XPUB descriptor. Looking at the wallet that Core created, XPUB/XPRIV descriptors contain derivation path while cormorant wallet does not. Maybe that is the problem?

[craigraw]

What's missing is the address in cormorant wallet that received transaction in step 9.

This is because the address is derived from the descriptor.

Core also doesn't show the transaction from step 9.

The Core cormorant wallet is part of the internal implementation of the Cormorant library - it is not intended to be used outside of Sparrow, and neither is such use supported.

While sparrow shows everything as expected, I am concerned that if I would recreate from backup the 2 wallets in Sparrow, it wouldn't fetch all transactions using Cormorant library, especially if Core is pruned.

So long as you have the seed, you can create the descriptor. So long as you have the descriptor, you can find all the transactions, in Sparrow or any other standards-compliant wallet. The internal implementation details of Cormorant aren't relevant to this process.

It appears we have moved some way from the original issue as titled here. Can we close this off?

[eugenesan]

Thank you for explaining.

I don't think we moved away from the original issue. Leaving aside all the technical details (which I admittedly might have got wrong) , I am still not clear on how to properly sync multiple Sparrow wallets with Cormorant to avoid "incomplete sync" issues?

Also, with pruned Core + Cormorant + multiple Sparrow wallets, will I have to re-download the whole Core DB if one of the Sparrow wallet files is lost?

[craigraw]

Leaving aside all the technical details (which I admittedly might have got wrong) , I am still not clear on how to properly sync multiple Sparrow wallets with Cormorant to avoid "incomplete sync" issues?

Best advice is to avoid stopping Core while it is scanning, which creates an incomplete scan Sparrow is not able to detect as it is not reported by the Core API.

Also, with pruned Core + Cormorant + multiple Sparrow wallets, will I have to re-download the whole Core DB if one of the Sparrow wallet files is lost?

No, the Sparrow wallet files aren't important with pruning - it's keeping the Core cormorant wallet open while Core is receiving new blocks. See here for more info: https://bitcoin.stackexchange.com/questions/88382/background-information-on-warning-when-closing-wallet-in-pruned-mode/88386#88386

[eugenesan]

Thank you very much.

One more question and we can close the issue.

What do you think about the following procedure to avoid multiple re-scans with multiple wallets:

1. Close the Core
2. Delete cormorant wallet in Core
3. Open Sparrow
4. Open/create multiple wallets
5. Start the Core
6. Connect Sparrow to the Core using Cormorant
7. Wait for the sync to complete
8. Prune the Core

[craigraw]

Yes, that should work fine.