ongrid commented 4 months ago

Trying to execute commit-reveal process for ordinals I noticed that after some size threshold the following code produces invalid raw transactions that get discarded by broadcasting endpoints with the following error: Error: non-mandatory-script-verify-flag (Witness program hash mismatch).

Context

Ordinals standard allows to store various metadata in taproot scripts on-chain https://docs.ordinals.com/inscriptions/metadata.html. Examples prove that there is

Expected Behavior

Here is the proper flow with downsized witness:

Reproduction

The code below has larger taproot script, and produces invalid reveal Tx

import json
from backend.bitcoin.brc20.operations import BRC20Operation, BRC20Deploy, BRC20Mint, BRC20Transfer
from backend.bitcoin.brc20.inscribe import BTCExecutor, TxInA
from backend.bitcoin.electrum_rpc_client import ElectrumRpcClient, select_random_server
from bitcoinutils.setup import setup
from bitcoinutils.transactions import Transaction, TxInput, TxOutput, TxWitnessInput
from bitcoinutils.hdwallet import HDWallet
from backend.bitcoin.util import get_script_hash
from bitcoinutils.script import Script
from bitcoinutils.utils import to_satoshis, ControlBlock
NETWORK = "testnet"
setup(NETWORK)
MNEMONIC=
hdw = HDWallet(mnemonic=MNEMONIC)
hdw.from_path("m/86'/0'/0'/0/0")
priv_key = hdw.get_private_key()
pub_key = priv_key.get_public_key()
sender = pub_key.get_taproot_address()

I'm trying to put HTML page into taproot script

script = """<script>
  window.onload = function() {
    var img = document.getElementById("myImage");
    if(img) {
      var a = document.createElement("a");
      a.href = "https://example.com";
      img.parentNode.insertBefore(a, img);
      a.appendChild(img);
    }
  }
</script>"""

base64_image="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAgAAAAIAQMAAAD+wSzIAAAABlBMVEX///+/v7+jQ3Y5AAAADklEQVQI12P4AIX8EAgALgAD/aNpbtEAAAAASUVORK5CYII"

html_content = f"""<!DOCTYPE html>
<html>
  <head>
    {script}
  </head>
  <body>
    <img src="{base64_image}" id="myImage">
  </body>
</html>
"""
html_content = ' '.join(html_content.split())
html_content = html_content.replace("> <", "><")

I encode the data into OP_IF - OP_ENDIF block

taproot_script = Script(
    [
        pub_key.to_x_only_hex(),
        "OP_CHECKSIG",
        "OP_0",
        "OP_IF",
        "ord".encode("utf-8").hex(),
        "01",
        "text/html;charset=utf-8".encode("utf-8").hex(),
        "OP_0",
        html_content.encode("utf-8").hex(),
        "OP_ENDIF",
    ]
)

Calculate fees and values for outputs

INSCRIPTION_SATS = 350
COMMIT_FEE = 500
sat_per_vbyte = 50.0
# calculate fee
legacy_part_len = 96
witness_len = 0
witness_len += 1  # WITNESS STACK LENGTH PER EACH WITNESS
witness_len += 64  # Schnorr Signature
witness_len += len(taproot_script.to_bytes())  # Variable part of inscriptions
witness_len += 33  # Control block
vsize = legacy_part_len + witness_len / 4
fee_sat = int(sat_per_vbyte * vsize)
fee_sat

Getting UTXO to spend (from the wallet API)

vin_to_spend = {'txid': 'bcf329648a74c93ffc95501fad36d0236045d7d42bbda9f328821cdf723d6c4e',
 'vout': 1,
 'satoshis': 199789,
 'scriptPk': '51201b72e9c6199724250c146793d1b72be8409afa6a52390ac004f7b54aacb38844',
 'addressType': 2,
 'inscriptions': [],
 'atomicals': [],
 'pubkey': '',
 'height': 2580530}

Generate "commit" Tx

vin = TxInput(vin_to_spend['txid'], vin_to_spend['vout'])
vout_commit = TxOutput(fee_sat + INSCRIPTION_SATS, taproot_script_address.to_script_pub_key())
vout_change = TxOutput(vin_to_spend['satoshis'] - fee_sat - INSCRIPTION_SATS - COMMIT_FEE, sender.to_script_pub_key())
commit_tx = Transaction([vin], [vout_commit, vout_change], has_segwit=True)
sig = priv_key.sign_taproot_input(commit_tx, 0, [sender.to_script_pub_key()], [vin_to_spend['satoshis']])
commit_tx.witnesses.append(TxWitnessInput([sig]))
commit_tx.serialize()

It's always properly broadcasted

Examples: 22cd4a7002b2e4e19abf5fa15694024c3fb3e80dbe4657c5630b708962ffe561

Building reveal Tx:

vin = TxInput(commit_tx_hash, 0)
vout = TxOutput(INSCRIPTION_SATS, recipient.to_script_pub_key())
reveal_tx = Transaction([vin], [vout], has_segwit=True)
sig = priv_key.sign_taproot_input(
    reveal_tx,
    0,
    [taproot_script_address.to_script_pub_key()],
    [commit_tx.outputs[0].amount],
    script_path=True,
    tapleaf_script=taproot_script,
    tapleaf_scripts=[taproot_script],
    tweak=False,
)
control_block = ControlBlock(pub_key)
reveal_tx.witnesses.append(
    TxWitnessInput(
        [sig, taproot_script.to_hex(), control_block.to_hex()]
    )
)
reveal_tx.serialize()

Nodes / broadcasting services return Error: non-mandatory-script-verify-flag (Witness program hash mismatch). The only service that accepts these Txes is https://live.blockcypher.com/btc/pushtx/ (but anyway, these transactions are never relayed to other nodes and never included in the block)

Examples of failing txes:

b707672166b71c86c85a608b8aec9db715892ec511a6a0752a95f509a65bba37

020000000001013f05e023453efb0ba9a11d0eff0db27cad57de1d87a2658424e09fd985159d1b0000000000ffffffff015e010000000000002251201b72e9c6199724250c146793d1b72be8409afa6a52390ac004f7b54aacb3884403403d95b246786d0962ee33dffa4f3bc9a022c97e77f7c35d13a8ba2b339cd1bfbac3dcdcb5aaa17cba0ad0a72116183d8adbc0176473e056be82a53a165a818c3efd0c02202bcb1f34b7427ab9b99ec272bf9dbe88b2a9049a38c9e7ebf2da8f3db2e4cd76ac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004dc5013c21444f43545950452068746d6c3e3c68746d6c3e3c686561643e3c7363726970743e2077696e646f772e6f6e6c6f6164203d2066756e6374696f6e2829207b2076617220696d67203d20646f63756d656e742e676574456c656d656e744279496428226d79496d61676522293b20696628696d6729207b207661722061203d20646f63756d656e742e637265617465456c656d656e7428226122293b20612e68726566203d202268747470733a2f2f6578616d706c652e636f6d223b20696d672e706172656e744e6f64652e696e736572744265666f726528612c20696d67293b20612e617070656e644368696c6428696d67293b207d207d203c2f7363726970743e3c2f686561643e3c626f64793e3c696d67207372633d22646174613a696d6167652f706e673b6261736536342c6956424f5277304b47676f414141414e5355684555674141414167414141414941514d414141442b77537a4941414141426c424d5645582f2f2f2b2f76372b6a5133593541414141446b6c45515651493132503441495838454167414c6741442f614e7062744541414141415355564f524b3543594949222069643d226d79496d616765223e3c2f626f64793e3c2f68746d6c3e6821c02bcb1f34b7427ab9b99ec272bf9dbe88b2a9049a38c9e7ebf2da8f3db2e4cd7600000000

5095c3caf4dc1a8d8b813596bb3f62b52b7ef59ec6e2b239a4d717038aef3660

0200000000010103df6c72f4a48cba6e318d1d20bf06d2a1660d4729af9d16148967f54cd1eed10000000000ffffffff015e010000000000002251201b72e9c6199724250c146793d1b72be8409afa6a52390ac004f7b54aacb3884403404b16a9d2df6dcc3e81e6e4381d14aaa516fabe827712ba9452a60c31691f7f5b5d7b0299661435f710ca9d0a21dd300cf4823be8bc3957c8111f0436bf1d75d8fd0c02202bcb1f34b7427ab9b99ec272bf9dbe88b2a9049a38c9e7ebf2da8f3db2e4cd76ac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004dc5013c21444f43545950452068746d6c3e3c68746d6c3e3c686561643e3c7363726970743e2077696e646f772e6f6e6c6f6164203d2066756e6374696f6e2829207b2076617220696d67203d20646f63756d656e742e676574456c656d656e744279496428226d79496d61676522293b20696628696d6729207b207661722061203d20646f63756d656e742e637265617465456c656d656e7428226122293b20612e68726566203d202268747470733a2f2f6578616d706c652e636f6d223b20696d672e706172656e744e6f64652e696e736572744265666f726528612c20696d67293b20612e617070656e644368696c6428696d67293b207d207d203c2f7363726970743e3c2f686561643e3c626f64793e3c696d67207372633d22646174613a696d6167652f706e673b6261736536342c6956424f5277304b47676f414141414e5355684555674141414167414141414941514d414141442b77537a4941414141426c424d5645582f2f2f2b2f76372b6a5133593541414141446b6c45515651493132503441495838454167414c6741442f614e7062744541414141415355564f524b3543594949222069643d226d79496d616765223e3c2f626f64793e3c2f68746d6c3e6821c02bcb1f34b7427ab9b99ec272bf9dbe88b2a9049a38c9e7ebf2da8f3db2e4cd7600000000

If I make content ~30% shorter, the reveal tx becomes valid (see "Expected behavior examples").

Highly appreciate any help

karask commented 4 months ago

Thanks for sharing. I will look into it as soon as I find some time.

Btw, up to approximately what size of witness does it stop working?

ongrid commented 4 months ago

I will reiterate later this week, check and come back with updates, but if you see any obvious misuse of the library and data or any other recommendations that may help in debugging, please let me know.

jonasmartin commented 4 months ago

@karask @ongrid I was facing a similar problem and I think it might be related with the sign bit on the control block. I pushed a PR that might help. (I actually could confirm modifying a little bit the example from @ongrid that the fix works) I'm kind of new on bitcoin protocols, so PTAL if it works for you.

ongrid commented 3 months ago

My experiments have shown that the result depends on two factors - both size AND content ind it seems two separate issues:

Repro

if payload is equal or greater 128 bytes (or entire taproot script is 200B+) it always fails.
if payload is 127B or less, the outcome depends on the content with equal distribution (result is deterministic if I provide the same data). Are this side effects expected with your findings #64 @jonasmartin ?

These payloads always succeed

payload = 'deadbeef' * 31 + "dead" + '83'
payload = 'deadbeef' * 31 + "dead" + '53'

These payloads always fail

payload = 'deadbeef' * 31 + "dead" + 'f3'
payload = 'deadbeef' * 31 + "dead" + '7b'

The failure reason is same in both cases - when I push raw tx to electrumx on this line I get

Exception: the transaction was rejected by network rules.
mandatory-script-verify-flag-failed (Witness program hash mismatch)

ongrid commented 3 months ago

@jonasmartin I've installed bitcoin-utils from your repo #64

pip install git+https://github.com/jonasmartin/python-bitcoin-utils.git
Collecting git+https://github.com/jonasmartin/python-bitcoin-utils.git
  Cloning https://github.com/jonasmartin/python-bitcoin-utils.git to /private/var/folders/31/zqqkzzk1645f56szy7pzvtym0000gn/T/pip-req-build-kb1ezchh
  Running command git clone --filter=blob:none --quiet https://github.com/jonasmartin/python-bitcoin-utils.git /private/var/folders/31/zqqkzzk1645f56szy7pzvtym0000gn/T/pip-req-build-kb1ezchh
  Resolved https://github.com/jonasmartin/python-bitcoin-utils.git to commit 8b8ea5c473bf9cbf2fd7f095d785086a1ebe81dd
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: base58check<2.0,>=1.0.2 in ./.venv/lib/python3.12/site-packages (from bitcoin-utils==0.6.5) (1.0.2)
Requirement already satisfied: ecdsa==0.17.0 in ./.venv/lib/python3.12/site-packages (from bitcoin-utils==0.6.5) (0.17.0)
Requirement already satisfied: sympy<2.0,>=1.2 in ./.venv/lib/python3.12/site-packages (from bitcoin-utils==0.6.5) (1.12)
Requirement already satisfied: python-bitcoinrpc<2.0,>=1.0 in ./.venv/lib/python3.12/site-packages (from bitcoin-utils==0.6.5) (1.0)
Requirement already satisfied: hdwallet==2.2.1 in ./.venv/lib/python3.12/site-packages (from bitcoin-utils==0.6.5) (2.2.1)
Requirement already satisfied: six>=1.9.0 in ./.venv/lib/python3.12/site-packages (from ecdsa==0.17.0->bitcoin-utils==0.6.5) (1.16.0)
Requirement already satisfied: mnemonic<1,>=0.19 in ./.venv/lib/python3.12/site-packages (from hdwallet==2.2.1->bitcoin-utils==0.6.5) (0.21)
Requirement already satisfied: pycryptodome<4,>=3.15 in ./.venv/lib/python3.12/site-packages (from hdwallet==2.2.1->bitcoin-utils==0.6.5) (3.20.0)
Requirement already satisfied: base58<3,>=2.0.1 in ./.venv/lib/python3.12/site-packages (from hdwallet==2.2.1->bitcoin-utils==0.6.5) (2.1.1)
Requirement already satisfied: mpmath>=0.19 in ./.venv/lib/python3.12/site-packages (from sympy<2.0,>=1.2->bitcoin-utils==0.6.5) (1.3.0)
Building wheels for collected packages: bitcoin-utils
  Building wheel for bitcoin-utils (pyproject.toml) ... done
  Created wheel for bitcoin-utils: filename=bitcoin_utils-0.6.5-py3-none-any.whl size=43509 sha256=cc9d44c476bdf7057118aa3adc82b3541abe6a89c9e9bc022338495d1e6785d7
  Stored in directory: /private/var/folders/31/zqqkzzk1645f56szy7pzvtym0000gn/T/pip-ephem-wheel-cache-5oki0vzy/wheels/3e/01/42/a6eb8003a9d836a567dcbd5b2071a95ba90d4db21727998fcf
Successfully built bitcoin-utils
Installing collected packages: bitcoin-utils
Successfully installed bitcoin-utils-0.6.5

The behavior is still same. "Lucky" payloads continue working, Failing - still fail.

jonasmartin commented 3 months ago

@ongrid when you tried my changes did you also modified your code? It requires a simple change when creating the controlblock like this: https://github.com/karask/python-bitcoin-utils/pull/64/files#diff-de03fd558138bb99805803f9139b364e3dc9462b1ab228dad615d6f7d2f53477R114 In your code the change should be here: https://github.com/ongrid/bitcoin-playground/blob/42a4405cee9bff4018ac1b4f1c2ea9b964987c5f/generator/inscribe.py#L122

ongrid commented 3 months ago

Tried after your comment @jonasmartin

-control_block = ControlBlock(alice_priv_key.get_public_key())
+control_block = ControlBlock(alice_priv_key.get_public_key(), is_odd=alice_p2tr.is_odd())

With random byte in payload, results are still sporadical

+import random
+random_byte = format(random.randint(0, 255), '02x')
+payload = 'deadbeef' * 31 + "dead" + random_byte

jonasmartin commented 3 months ago

@ongrid Pleaes try with taproot_script_address.is_odd() because is_odd result is dependent of the script passed as argument when calculating tweaking

ongrid commented 3 months ago

@jonasmartin fantastic! It works 🎉. Both size limitations and content issues solved by #64! Highly apreciate!

jonasmartin commented 3 months ago

Oh great news! 👍

karask commented 3 months ago

@ongrid , @jonasmartin solution was fine. I just merged his PR; new version is v0.6.6. Also updated library in pypi. Thanks for reporting!

karask / python-bitcoin-utils

Large typeroot script leads to Error: non-mandatory-script-verify-flag (Witness program hash mismatch) #63

Context

Expected Behavior

Reproduction

Examples of failing txes: