Unable to derive output scripts for hardened child indexes of a descriptor

[notmandatory]

This may just be a question rather than a bug; is there a way to use the descriptor! macro to create a descriptor from which I can derive output scripts for a hardened child index? The below test demonstrates the problem. I don't know if I'm just doing something wrong with the keys, or if this is not a supported feature. The particular test vector I'm using is copied from: https://github.com/bitcoin/bitcoin/blob/master/src/test/descriptor_tests.cpp.

    #[test]
    fn hardened_idx_test() {
        let xprv = bip32::ExtendedPrivKey::from_str("xprv9s21ZrQH143K3QTDL4LXw2F7HEK3wJUD2nW2nRk4stbPy6cq3jPPqjiChkVvvNKmPGJxWUtg6LnF5kejMRNNU3TGtRBeJgk33yuGBxrMPHi").unwrap();
        let path = bip32::DerivationPath::from_str("m/10/20/30/40").unwrap();
        let prvkey = (xprv, path.clone())
            .to_descriptor_key()
            .unwrap();

        println!("prvkey = {:?}", prvkey);

        let (desc, keymap, networks) = descriptor!(sh(wpkh(prvkey))).unwrap();

        println!("desc = {:?}", desc);

        // This also fails with a parsing error.
        //
        // let desc2 = Descriptor::<DescriptorPublicKey>::from_str(
        //     "sh(wpkh(xprv9s21ZrQH143K3QTDL4LXw2F7HEK3wJUD2nW2nRk4stbPy6cq3jPPqjiChkVvvNKmPGJxWUtg6LnF5kejMRNNU3TGtRBeJgk33yuGBxrMPHi/10/20/30/40/*'))",
        // ).unwrap();
        //
        // println!("desc2 = {:?}", desc2);

        let script = desc.derive(&[ChildNumber::from_hardened_idx(0).unwrap()])
            .script_pubkey();

        println!("script = {:?}", script);
    }

outputs:

prvkey = Secret(XPrv(DescriptorXKey { source: None, xkey: ExtendedPrivKey { network: bitcoin, depth: 0, parent_fingerprint: Fingerprint([0, 0, 0, 0]), child_number: Normal { index: 0 }, private_key: [private key data], chain_code: ChainCode([135, 61, 255, 129, 192, 47, 82, 86, 35, 253, 31, 229, 22, 126, 172, 58, 85, 160, 73, 222, 61, 49, 75, 180, 46, 226, 39, 255, 237, 55, 213, 8]) }, derivation_path: m/10/20/30/40, is_wildcard: true }), {bitcoin}, PhantomData)

desc = sh(wpkh(XPub(DescriptorXKey { source: None, xkey: ExtendedPubKey { network: bitcoin, depth: 0, parent_fingerprint: Fingerprint([0, 0, 0, 0]), child_number: Normal { index: 0 }, public_key: PublicKey { compressed: true, key: PublicKey(c2859cf48f2e0e05f12dec2755b5d013c52ca093e5fb41efda9715301360a3398162110073e301bf42eae2051382c8c5db8f88628f6b8994a5e67c0eed7dbe3c) }, chain_code: ChainCode([135, 61, 255, 129, 192, 47, 82, 86, 35, 253, 31, 229, 22, 126, 172, 58, 85, 160, 73, 222, 61, 49, 75, 180, 46, 226, 39, 255, 237, 55, 213, 8]) }, derivation_path: m/10/20/30/40, is_wildcard: true })))

error:

assertion failed: path.into_iter().all(|c| c.is_normal())
thread 'descriptor::dsl::test::hardened_idx_test' panicked at 'assertion failed: path.into_iter().all(|c| c.is_normal())', /home/steve/.cargo/git/checkouts/rust-miniscript-513a889ddeb16d56/d0322ac/src/descriptor/mod.rs:410:9
stack backtrace:

expected script (hex): a9149a4d9901d6af519b2a23d4a2f51650fcba87ce7b87

[afilini]

That's not a bug, it's the expected behavior, but it's definitely a feature that we should try to add going forward.

This is due to the way rust-miniscript handles, or rather does not handle, secret keys: since miniscript itself doesn't need to know anything about secrets, it was decided to only store public keys inside descriptors and keep private keys in a separate map. When a descriptor with secret keys is parsed using Descriptor::parse_secret() it is returned as a tuple (Descriptor<DescriptorPublicKey>, KeyMap). This is also a good generalization for external signers, like hardware wallets: in the descriptor we only place the public key and then the handling of private keys or other signers is handled externally.

However, this means that miniscript doesn't support descriptors with an "hardened wildcard" at the end, because it would need to know the secret to make that derivation.

I think the right approach here would probably be to extend the Signer trait and add a method to request the derivation of a key with a given path, and then either make changes directly to rust-miniscript to somehow integrate that, or only try to work around it in our code. That approach would be the most "generalized", and it would also work with external signers and not only keys for which miniscript has the secret.

[notmandatory]

As discussed in team chat today, bdk (via rust-miniscript) would need to support "hardened wildcard" descriptors to be able to import any wallet from core, which supports this feature.

[ghscuuo]

Any updates on this feature, or any alternatives or workarounds?

For example, how would I spend these 10000 tsats? https://mempool.space/testnet/address/tb1qpwry422kc07xsfcem0mvkkyq50xq0deunr8ugn

Account Extended Private Key vprv9LL3fRJ1DH6mDqcovVdpjcEQqn9d8xJko28ti2SqmiNwpg1dkQRftjho2mKBbPSM4SxqLFbowXWcDWecqWiVaHunXSuZLKsa8R8RYg1j4nx

Account Extended Public Key vpub5ZKQ4vpu3ef4SKhH2XAq6kB9Poz7YR2cAF4VWQrTL3uvhULnHwjvSY2Gt3Z7XAaYhskpyfDuYBD639i4DKf3RRjjZ5rsNyJ5ZYwC18mYT9s

BIP32 Derivation Path m/84'/1'/0'/0

BIP32 Extended Private Key vprv9M2VjFRSEafsiNCDZR8vR6hp6eqH5VE5u4vpdrDbhgrbvXYsm62RiDUCZ6k8EqoJjewR44n84r58Gyz87kR12R5hijxGDrBFEvatWD1PvQJ

BIP32 Extended Public Key vpub5a1r8kxL4xEAvrGgfSfvnEeYegfmUwwwGHrRSEdDG2PaoKt2JdLgG1ngQN6N186dDYzRSCw7wXGrmqQwbx5gZqwL4zSkZU9qf4MS99SmX1d

Derived Addresses x Use hardened addresses m/84'/1'/0'/0/0' tb1qpwry422kc07xsfcem0mvkkyq50xq0deunr8ugn Public key: 03a9801545dbc006a30e09535115aa5c1429114fa2774e44b26dab32f5bef952c0 Private key: cN7HBuYG3p1xtXUH4EngdboeXqeXZ8Xww8m5vjNGHUbSEkHxiQhP

Credit: https://github.com/iancoleman/bip39/releases/tag/0.5.4