Spend from P2SH-P2WPKH endorsement produces "Witness program hash mismatch" rejection

[SixFiveSoftware]

Firstly, thank you for your time and effort in producing such a tremendous framework. We are using it to create native iOS and Android versions of our app. To that point, both iOS and Android are experiencing random rejections from nodes after broadcasting a transaction via an obelisk_client. Not all nodes reject the transactions, in fact, most go through ok, and also pushing the raw transaction to blockchain.info's pushtx endpoint proves that it's a usable transaction since these outputs are always spendable in future transactions, these rejections are worrisome. Creating the exact same transaction using the same inputs and outputs, same values, etc in Electrum provides the exact same encoded transaction except for the witness endorsement signature.

Has anyone else experienced this?

The error returned from the nodes in full is: [network] Received tx reject (64) from [18.210.109.9:8333] 'non-mandatory-script-verify-flag (Witness program hash mismatch)' [112c6b57c78c8b34ae0b3b373f4ae9f8e492eb3bd2b3f2d71d8f70dc7478f052]

The error triggered appears to be from here, as I have verified that our witness stack size is 2, so it is not the other case that triggers it.

Constructing the transaction, we are providing the proper UTXOs, signing from proper private keys, and so on, but for some reason, it generates a different witness signature than Electrum does. Perhaps it's ok and doesn't need to match? But one would think it should. The code we used is identical to Libbitcoin's wiki for spending from P2SH(P2WPKH), as well as the examples given by Aaron Jaramillo's tutorials. I am happy to provide any other code we use if it will help.

Is there a more nuanced way to sign the inputs used in a transaction than what the documentation states? Or is it just an issue of the nodes listening on the network that aren't equipped to handle P2SH-P2WPKH transactions? I have been digging around for days trying to figure out why this is happening, but have come to the point where I am stumped and wanted to reach out to the community.

Thank you in advance for your consideration.

[evoskuil]

Hi @SixFiveSoftware, thanks for the feedback. How have you determined that some nodes accept the invalid witness tx?

[SixFiveSoftware]

Thanks @evoskuil , some nodes accept it because the transactions will be added to the mempool and subsequently mined into a block later on, as long as at least one listening node doesn't reject it. We spun up our own Libbitcoin server instance too, so we could analyze the logs when broadcasting, and that's how we discovered the aforementioned rejection notice.

[SixFiveSoftware]

One example:

Generated by us: 01000000000101c2fa1f3dfcc8d121f423885bfbb4579b1f1b653b7b7b86393aeddde154b58b9a010000001716001443ce1f475df701b3e26007578619054fadf75720feffffff02a08601000000000017a914749ff43374f3c2d9ffa2fe239dc5a805b9599e198736c800000000000017a91474fe94b2cbf0d60787ec5db8ffb8741cf95496f8870248304502210097c2e0816f83ffd3bc95ce102156bc3202906d45744f9b2e5bef7dc74eb8847e022061d9fcb88860a9bfa28ffcf65c9612180fffcc52601830c803a417f0f0300a3d0121033fde0485b998f456fa20a38ac73666f129fe07192d7b56246b7074c8c02048409e400800

Generated by Electrum with same inputs/outputs: 01000000000101c2fa1f3dfcc8d121f423885bfbb4579b1f1b653b7b7b86393aeddde154b58b9a010000001716001443ce1f475df701b3e26007578619054fadf75720feffffff0236c800000000000017a91474fe94b2cbf0d60787ec5db8ffb8741cf95496f887a08601000000000017a914749ff43374f3c2d9ffa2fe239dc5a805b9599e198702483045022100d2306b52a518c1c9dcfcc7842b3288c70c7a9691c80d03c5b8eaeafe20f200550220274ae5504ff0d48fd4f2cabd621721a6c34d7e8af3a75444f56a6d5190631e570121033fde0485b998f456fa20a38ac73666f129fe07192d7b56246b7074c8c02048409e400800

[evoskuil]

That doesn’t really answer the question. How do you differentiate between acceptances and rejections?

[evoskuil]

You cannot rely on reject messages from nodes. Some nodes don’t send them. In other words, it seems likely that your tx is invalid and is being rejected by all peers. Also, if a node doesn’t support segwit then the segwit-serialized tx will appear to have no inputs, and be treated as invalid. And you are also getting invalid segwit tx notices. So the tx is segwit and presumably always invalid.

[evoskuil]

The reject message is pretty specific. If you have the same inputs and outputs in the same order, and all other fields of the tx are the same, and you sign with the same key(s) with the same sighash then the tx serialization must be the same. So you just appear to be constructing the tx incorrectly.

[SixFiveSoftware]

Oh, sorry. With respect to our transaction makeup, we couldn't identify. We couldn't discern why some would reject and some would accept. Some transactions would have nothing in the logs, some might have (from what we've seen) anywhere between 1 to 6 rejection messages from peers.

[SixFiveSoftware]

If the tx is constructed incorrectly, how is it spendable as a UTXO in future transactions? Even if I send to self, so I control the subsequent signing keys, I'm still able to use these outputs. I don't doubt that it could certainly be something I'm doing, but as mentioned before, I'm following the tutorials and documentation to the closest degree I can find possible, and cannot understand why the signature is different (yet still can be successfully broadcast and mined).

[evoskuil]

Are you saying that the transaction is getting confirmed in a block?

[SixFiveSoftware]

Yes sir. A transaction I sent that had four rejection messages, but still was mined was 473206c4908e27c30a6d3331c4cea261afda14e1e9826ce01cf089ed8627a07c

And from the server logs...

$ grep 473206c4908e27c30a6d3331c4cea261afda14e1e9826ce01cf089ed8627a07c archive/*.log archive/debug-00813.log:11:46:06.914316 VERBOSE [server] Published public transaction [473206c4908e27c30a6d3331c4cea261afda14e1e9826ce01cf089ed8627a07c] (61480).

archive/debug-00813.log:11:46:07.280452 DEBUG [network] Received tx reject (64) from [52.48.18.13:8333] 'non-mandatory-script-verify-flag (Witness program hash mismatch)' [473206c4908e27c30a6d3331c4cea261afda14e1e9826ce01cf089ed8627a07c].

archive/debug-00813.log:11:46:07.348864 DEBUG [network] Received tx reject (64) from [122.100.186.58:8333] 'non-mandatory-script-verify-flag (Witness program hash mismatch)' [473206c4908e27c30a6d3331c4cea261afda14e1e9826ce01cf089ed8627a07c].

archive/debug-00813.log:11:46:07.450682 DEBUG [network] Received tx reject (64) from [47.75.149.195:8333] 'non-mandatory-script-verify-flag (Witness program hash mismatch)' [473206c4908e27c30a6d3331c4cea261afda14e1e9826ce01cf089ed8627a07c].

archive/debug-00813.log:11:46:07.451109 DEBUG [network] Received tx reject (64) from [222.239.10.164:8333] 'non-mandatory-script-verify-flag (Witness program hash mismatch)' [473206c4908e27c30a6d3331c4cea261afda14e1e9826ce01cf089ed8627a07c].

[evoskuil]

Oh, this is a standardness test:

                } else if (!check()) {
                    if (flags & STANDARD_NOT_MANDATORY_VERIFY_FLAGS) {
                        // Check whether the failure was caused by a
                        // non-mandatory script verification check, such as
                        // non-standard DER encodings or non-null dummy
                        // arguments; if so, don't trigger DoS protection to
                        // avoid splitting the network between upgraded and
                        // non-upgraded nodes.
                        CScriptCheck check2(scriptPubKey, amount, tx, i,
                                flags & ~STANDARD_NOT_MANDATORY_VERIFY_FLAGS, cacheSigStore, &txdata);
                        if (check2())
                            return state.Invalid(false, REJECT_NONSTANDARD, strprintf("non-mandatory-script-verify-flag (%s)", ScriptErrorString(check.GetScriptError())));
                    }

So rejection depends on node policy configuration.

[SixFiveSoftware]

Hey @evoskuil , thanks for the info and my apologies in the delay getting back on this. I wanted to spend some time to rewrite the transaction creation according to the documentation just to see if it was something I had inadvertently misconfigured. After recreating it, I was able to get the witness hashes in my generated transaction and the same transaction generated in Electrum to match. Interestingly enough, however, is that when broadcasting, some nodes still rejected with the same "witness program hash mismatch" message. So it seems to back what you are saying, in that some nodes may just not be configured with segwit abilities.

Out of curiosity, have you or anyone else ever come across this error in your experience with spending BIP49 P2SH-P2WPKH outputs? It would only be visible from a Libbitcoin server log file.

Thank you again for your time and consideration on this, it is much appreciated.

[evoskuil]

some nodes still rejected with the same "witness program hash mismatch" message. So it seems to back what you are saying, in that some nodes may just not be configured with segwit abilities.

A node would not reject with a witness error unless it was segwit-enabled.

Out of curiosity, have you or anyone else ever come across this error in your experience with spending BIP49 P2SH-P2WPKH outputs? It would only be visible from a Libbitcoin server log file.

Actually the error is coming from peer nodes, so it would be visible from any node that logged the reject message.

[SixFiveSoftware]

Sorry, I misspoke. I should have said "it is easily visible from a Libbitcoin server log file if you control a Libbitcoin server".

As for my question, have you ever seen that message?

[evoskuil]

No, but I haven’t looked.

[evoskuil]

Closing for inactivity.