search

karask / python-bitcoin-utils

Library to interact with the Bitcoin network. Ideal for low-level learning and experimenting.
MIT License
262 stars 99 forks source link

mandatory-script-verify-flag-failed (Push value size limit exceeded) error occurs when sending raw tx with Taproot inscriptions exceeding ~500KB #69

Open ongrid opened 2 months ago

ongrid commented 2 months ago

Update: The root cause of the mandatory-script-verify-flag-failed (Push value size limit exceeded) error when processing the sendrawtransaction command by Bitcoin Core, is the script tokensize limit defined in bitcoind MAX_SCRIPT_ELEMENT_SIZE = 520 constant https://github.com/bitcoin/bitcoin/blob/0de63b8b46eff5cda85b4950062703324ba65a80/src/script/script.h#L27

If you encounter this error, you'll need to break down long data into smaller tokens (I used 512 bytes for better kilobyte-alignment), and then use these minimized tokens in the script. Here’s a function and an example of how to use it, shown below.

# To avoid `mandatory-script-verify-flag-failed (Push value size limit exceeded)` in bitcoin-core
# See https://github.com/bitcoin/bitcoin/blob/0de63b8b46eff5cda85b4950062703324ba65a80/src/script/script.h#L27
# and https://github.com/karask/python-bitcoin-utils/issues/69
def chunk_script_element(data: bytes):

    BTC_CORE_MAX_SCRIPT_ELEMENT_SIZE = 512

    for i in range(0, len(data), BTC_CORE_MAX_SCRIPT_ELEMENT_SIZE):
        yield data[i:i+BTC_CORE_MAX_SCRIPT_ELEMENT_SIZE]

Example use for ordinals case

            taproot_script_p2pk = Script(
                [
                    priv_key.get_public_key().to_x_only_hex(),
                    "OP_CHECKSIG",
                    "OP_0",
                    "OP_IF",
                    "ord".encode("utf-8").hex(),
                    "01",
                    MIME_TYPE.encode("utf-8").hex(),
                    "OP_0",
                    *[chunk.hex() for chunk in chunk_script_element(content)],
                    "OP_ENDIF",
                ]
            )

My suspicions that the problem was with the functions in the Script class were not confirmed.

Script.to_bytes() function and especially _op_push_data seems to populate OP_PUSHDATA_2 length padding using incorrect endianness when handling scripts containing hexadecimal data exceeding 76 bytes.

This may lead to various issues with script execution like mandatory-script-verify-flag-failed (Opcode missing or not understood) on broadcasting raw transaction. The to_hex() and from_raw() serialization functions are exhibiting unreliable behavior when handling scripts containing hexadecimal data exceeding 76 bytes. This issue disrupts the consistency of data posted by PUSHDATA2, leading to padding errors.

I noticed the incorrect length when I investigated errors of "Push value size limit exceeded":

mandatory-script-verify-flag-failed (Push value size limit exceeded)
[02000000000101c076929da8ada601dd2d16ce6ce269df47b86c23914fb56cfdc9eaede9c93e6c0000000000ffffffff025e01000000000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d32cf030000000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d03404af188ce6dfda88a894e1cd63f4949f5957095e9eb552932a38e67eb51e5f22aadf4352d4c05f95dc7a5bdd2387f38ba432e0a7b1bfdcac888f239173fdf9b9dfd530220c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebcaac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004d0c023c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f706e673b6261736536342c6956424f5277304b47676f414141414e5355684555674141414251414141415543414d4141414336562b302f4141414141584e535230494232636b736677414141416c7753466c7a4141414c4577414143784d42414a71634741414141476c51544652462f2f2f2f33392f663976623255564652374f7a736934754c34754c69336433644f44673465586c35385044775a47526b32646e5a7036656e2b767236364f6a6f484277636d4a6959585631644c437773626d35753565586c6b5a475250543039684953452f66333953456849666e352b73374f7a6348427778736247304e44516f4b436775377537546b354f46686278484141414148464a52454655654a79396a306b4b6744414d5257506256447659326a7250772f305061525158376755666849524838694541662b4554676c454a594b76734a5530536369775632684a78742b33574c614572734b46646e6f4c6731484e3158624c734472686b77666d6b597a757a747a796336796d2f4764397971477552704a6f53486f6b656444516d324d705547707a36376648506e44484242484774546235794141414141456c46546b5375516d4343223e3c2f626f64793e3c2f68746d6c3e6821c0c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebca00000000]

When a script is created with hex data longer than 76 bytes, the PUSHDATA2 length bytes are not handled correctly:

Reproduction

privkey_x_only_hex = "de" * 31
content = "deadbeef" * 100
script = Script(
    [
        privkey_x_only_hex,
        "OP_CHECKSIG",
        "OP_0",
        "OP_IF",
        content,
        "OP_ENDIF",
    ]
)
hex_dump = script.to_hex()

hex_dump value:

1fdededededededededededededededededededededededededededededededeac00634d9001deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef68

Actual content's datalength is 400B

>len(bytes.fromhex(content))
400
>hex(len(bytes.fromhex(content)))
'0x190'

('0x190' in hex)

_op_push_data generates the following sequence to wrap hex content setting length to hex(9001) that has decimal value of 36865.

4d    9001     deadbeefdeadbeef....
--    -----    ------------------
|      |          Data payload
|      | Length
|
OP_PUSHDATA2

PUSHDATA2 spec illustrates datalength should be formatted in normal (big-endian) order:

Examples: 0x4D 0x0100 <256 byte data item> - would leave the 256 byte data item on the stack.

So in our case, it should be

4d    0190     deadbeefdeadbeef....

Such misalignment (when length > max push size) seems to lead to the following trace on execution (example from live tx)

mandatory-script-verify-flag-failed (Push value size limit exceeded)
[02000000000101c076929da8ada601dd2d16ce6ce269df47b86c23914fb56cfdc9eaede9c93e6c0000000000ffffffff025e01000000000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d32cf030000000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d03404af188ce6dfda88a894e1cd63f4949f5957095e9eb552932a38e67eb51e5f22aadf4352d4c05f95dc7a5bdd2387f38ba432e0a7b1bfdcac888f239173fdf9b9dfd530220c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebcaac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004d0c023c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f706e673b6261736536342c6956424f5277304b47676f414141414e5355684555674141414251414141415543414d4141414336562b302f4141414141584e535230494232636b736677414141416c7753466c7a4141414c4577414143784d42414a71634741414141476c51544652462f2f2f2f33392f663976623255564652374f7a736934754c34754c69336433644f44673465586c35385044775a47526b32646e5a7036656e2b767236364f6a6f484277636d4a6959585631644c437773626d35753565586c6b5a475250543039684953452f66333953456849666e352b73374f7a6348427778736247304e44516f4b436775377537546b354f46686278484141414148464a52454655654a79396a306b4b6744414d5257506256447659326a7250772f305061525158376755666849524838694541662b4554676c454a594b76734a5530536369775632684a78742b33574c614572734b46646e6f4c6731484e3158624c734472686b77666d6b597a757a747a796336796d2f4764397971477552704a6f53486f6b656444516d324d705547707a36376648506e44484242484774546235794141414141456c46546b5375516d4343223e3c2f626f64793e3c2f68746d6c3e6821c0c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebca00000000]

When length is less than max_push_size, the cursor points to a byte of data that is treated as an opcode, resulting in a variety of OP-specific errors.

Preliminary root cause

My research show that root cause may be in the _op_push_data parser here, especially in pack formatter and direction of endianness:

https://github.com/karask/python-bitcoin-utils/blob/b10f493be4b3e2eadd9d91bf74e9cd04120a9f41/bitcoinutils/script.py#L302C1-L307C1

and fix can be as obvious as

 -           return b"\x4d" + struct.pack("<H", len(data_bytes)) + data_bytes
 +          return b"\x4d" + struct.pack(">H", len(data_bytes)) + data_bytes

But I'm unable to verify this solution due to another issue #68

ongrid commented 2 months ago

The issue wasn't confirmed.

I verified the script with bitcoin core decodescript JSON RPC API and it returned correctly parsed synthetic data I referenced above. This means serialization, padding and endianness are correct.

data = "1fdededededededededededededededededededededededededededededededeac00634d9001deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef68"
res = btc.call("decodescript", data).json()
print(res)
{'result': {'asm': 'dedededededededededededededededededededededededededededededede '
                   'OP_CHECKSIG 0 OP_IF '
                   'deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef '
                   'OP_ENDIF',
            'desc': 'raw(1fdededededededededededededededededededededededededededededededeac00634d9001deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef68)#gffl6knf',
            'type': 'nonstandard',
            'p2sh': '2N4eaM3Xii29TUbCAiNm8QDeHXGxg4JFJqF',
            'segwit': {'asm': '0 '
                              '642b23a4b031c501a5f990a772d5fc861e5fb21e84e376a0a1d3deb3f5f7391d',
                       'desc': 'addr(bcrt1qvs4j8f9sx8zsrf0ejznh940usc09lvs7sn3hdg9p600t8a0h8yws76hh5t)#y5q4pghe',
                       'hex': '0020642b23a4b031c501a5f990a772d5fc861e5fb21e84e376a0a1d3deb3f5f7391d',
                       'address': 'bcrt1qvs4j8f9sx8zsrf0ejznh940usc09lvs7sn3hdg9p600t8a0h8yws76hh5t',
                       'type': 'witness_v0_scripthash',
                       'p2sh-segwit': '2NC3Qnm8JCPqv2U1ydefXykm8W8pCaTxak3'}},
 'error': None,
 'id': 0}

I'm investigating further why my specific data is not processed properly though.

My script:

content = '<!DOCTYPE html><html><body><img width=100 height=100 src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAMAAAC6V+0/AAAAAXNSR0IB2cksfwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAGlQTFRF////39/f9vb2UVFR7Ozsi4uL4uLi3d3dODg4eXl58PDwZGRk2dnZp6en+vr66OjoHBwcmJiYXV1dLCwsbm5u5eXlkZGRPT09hISE/f39SEhIfn5+s7OzcHBwxsbG0NDQoKCgu7u7Tk5OFhbxHAAAAHFJREFUeJy9j0kKgDAMRWPbVDvY2jrPw/0PaRQX7gUfhIRH8iEAf+ETglEJYKvsJU0SciwV2hJxt+3WLaErsKFdnoLg1HN1XbLsDrhkwfmkYzuztzyc6ym/Gd9yqGuRpJoSHokedDQm2MpUGpz67fHPnDHBBHGtTb5yAAAAAElFTkSuQmCC"></body></html>'.encode("utf-8")

taproot_script_p2pk = Script(
    [
        priv_key.get_public_key().to_x_only_hex(),
        "OP_CHECKSIG",
        "OP_0",
        "OP_IF",
        "ord".encode("utf-8").hex(),
        "01",
        MIME_TYPE.encode("utf-8").hex(),
        "OP_0",
        content.hex(),
        "OP_ENDIF",
    ]
)
mandatory-script-verify-flag-failed (Push value size limit exceeded)
[0200000000010272514d229779f7072e697e0da24d3bc355bce1eba7fa1feb85b84021b8411b770000000000ffffffffc07ed4494c3f13e55b6f55d26cf1dcfefc50ea330d92f1165ce494fa62efe6ae0000000000ffffffff024605000000000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3df0ca052a01000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d0140340be8f174b01e5297a38c0c18eba275d746cc2a1e5dc631b66a86f67539ca066299f606c777cb08952349bb1bca74b5b5742baf7b2a78b5fe03a6b7af00851003401d4ee538bbf6b542416e93a512f5a55f5e4863ffada2914fb84bf01bc1668cd1b9b02eefbc0a409f957bd8a0954bd3d03c297f9384cedf505c573457d85bcf8dfd530220c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebcaac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004d0c023c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f706e673b6261736536342c6956424f5277304b47676f414141414e5355684555674141414251414141415543414d4141414336562b302f4141414141584e535230494232636b736677414141416c7753466c7a4141414c4577414143784d42414a71634741414141476c51544652462f2f2f2f33392f663976623255564652374f7a736934754c34754c69336433644f44673465586c35385044775a47526b32646e5a7036656e2b767236364f6a6f484277636d4a6959585631644c437773626d35753565586c6b5a475250543039684953452f66333953456849666e352b73374f7a6348427778736247304e44516f4b436775377537546b354f46686278484141414148464a52454655654a79396a306b4b6744414d5257506256447659326a7250772f305061525158376755666849524838694541662b4554676c454a594b76734a5530536369775632684a78742b33574c614572734b46646e6f4c6731484e3158624c734472686b77666d6b597a757a747a796336796d2f4764397971477552704a6f53486f6b656444516d324d705547707a36376648506e44484242484774546235794141414141456c46546b5375516d4343223e3c2f626f64793e3c2f68746d6c3e6821c0c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebca00000000]

Decoded tx:

{'result': {'txid': '8e9f1540396519d319044639cc539bc5c28ec5097c4ed08538203fbe7341fd94',
            'hash': 'd3c5559be1a14f5cf08a8fb8969ad9a3f23e3a8c419a86e1dfb1246c847221f6',
            'version': 2,
            'size': 944,
            'vsize': 370,
            'weight': 1478,
            'locktime': 0,
            'vin': [{'txid': '771b41b82140b885eb1ffaa7ebe1bc55c33b4da20d7e692e07f77997224d5172',
                     'vout': 0,
                     'scriptSig': {'asm': '', 'hex': ''},
                     'txinwitness': ['340be8f174b01e5297a38c0c18eba275d746cc2a1e5dc631b66a86f67539ca066299f606c777cb08952349bb1bca74b5b5742baf7b2a78b5fe03a6b7af008510'],
                     'sequence': 4294967295},
                    {'txid': 'aee6ef62fa94e45c16f1920d33ea50fcfedcf16cd2556f5be5133f4c49d47ec0',
                     'vout': 0,
                     'scriptSig': {'asm': '', 'hex': ''},
                     'txinwitness': ['1d4ee538bbf6b542416e93a512f5a55f5e4863ffada2914fb84bf01bc1668cd1b9b02eefbc0a409f957bd8a0954bd3d03c297f9384cedf505c573457d85bcf8d',
                                     '20c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebcaac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004d0c023c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f706e673b6261736536342c6956424f5277304b47676f414141414e5355684555674141414251414141415543414d4141414336562b302f4141414141584e535230494232636b736677414141416c7753466c7a4141414c4577414143784d42414a71634741414141476c51544652462f2f2f2f33392f663976623255564652374f7a736934754c34754c69336433644f44673465586c35385044775a47526b32646e5a7036656e2b767236364f6a6f484277636d4a6959585631644c437773626d35753565586c6b5a475250543039684953452f66333953456849666e352b73374f7a6348427778736247304e44516f4b436775377537546b354f46686278484141414148464a52454655654a79396a306b4b6744414d5257506256447659326a7250772f305061525158376755666849524838694541662b4554676c454a594b76734a5530536369775632684a78742b33574c614572734b46646e6f4c6731484e3158624c734472686b77666d6b597a757a747a796336796d2f4764397971477552704a6f53486f6b656444516d324d705547707a36376648506e44484242484774546235794141414141456c46546b5375516d4343223e3c2f626f64793e3c2f68746d6c3e68',
                                     'c0c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebca'],
                     'sequence': 4294967295}],
            'vout': [{'value': 1.35e-05,
                      'n': 0,
                      'scriptPubKey': {'asm': '1 '
                                              'a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d',
                                       'desc': 'rawtr(a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d)#vm7gj0as',
                                       'hex': '5120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d',
                                       'address': 'bcrt1p5um09qqgggcxtkm30r6xf602qxw5f5u47j375jrstm9h06ffts7s6t55e6',
                                       'type': 'witness_v1_taproot'}},
                     {'value': 49.9999,
                      'n': 1,
                      'scriptPubKey': {'asm': '1 '
                                              'a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d',
                                       'desc': 'rawtr(a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d)#vm7gj0as',
                                       'hex': '5120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d',
                                       'address': 'bcrt1p5um09qqgggcxtkm30r6xf602qxw5f5u47j375jrstm9h06ffts7s6t55e6',
                                       'type': 'witness_v1_taproot'}}]},
 'error': None,
 'id': 0}

Decoded script:

 {'result': {'asm': 'c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebca '
                   'OP_CHECKSIG 0 OP_IF 6582895 1 '
                   '746578742f68746d6c3b636861727365743d7574662d38 0 '
                   '3c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f706e673b6261736536342c6956424f5277304b47676f414141414e5355684555674141414251414141415543414d4141414336562b302f4141414141584e535230494232636b736677414141416c7753466c7a4141414c4577414143784d42414a71634741414141476c51544652462f2f2f2f33392f663976623255564652374f7a736934754c34754c69336433644f44673465586c35385044775a47526b32646e5a7036656e2b767236364f6a6f484277636d4a6959585631644c437773626d35753565586c6b5a475250543039684953452f66333953456849666e352b73374f7a6348427778736247304e44516f4b436775377537546b354f46686278484141414148464a52454655654a79396a306b4b6744414d5257506256447659326a7250772f305061525158376755666849524838694541662b4554676c454a594b76734a5530536369775632684a78742b33574c614572734b46646e6f4c6731484e3158624c734472686b77666d6b597a757a747a796336796d2f4764397971477552704a6f53486f6b656444516d324d705547707a36376648506e44484242484774546235794141414141456c46546b5375516d4343223e3c2f626f64793e3c2f68746d6c3e '
                   'OP_ENDIF',
            'desc': 'raw(20c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebcaac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004d0c023c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f706e673b6261736536342c6956424f5277304b47676f414141414e5355684555674141414251414141415543414d4141414336562b302f4141414141584e535230494232636b736677414141416c7753466c7a4141414c4577414143784d42414a71634741414141476c51544652462f2f2f2f33392f663976623255564652374f7a736934754c34754c69336433644f44673465586c35385044775a47526b32646e5a7036656e2b767236364f6a6f484277636d4a6959585631644c437773626d35753565586c6b5a475250543039684953452f66333953456849666e352b73374f7a6348427778736247304e44516f4b436775377537546b354f46686278484141414148464a52454655654a79396a306b4b6744414d5257506256447659326a7250772f305061525158376755666849524838694541662b4554676c454a594b76734a5530536369775632684a78742b33574c614572734b46646e6f4c6731484e3158624c734472686b77666d6b597a757a747a796336796d2f4764397971477552704a6f53486f6b656444516d324d705547707a36376648506e44484242484774546235794141414141456c46546b5375516d4343223e3c2f626f64793e3c2f68746d6c3e68)#5ww3yrlk',
            'type': 'nonstandard'},
 'error': None,
 'id': 0}

I successfully recovered my data in reverse direction: unaccepted rawtx -> JSON above -> script -> push2 argument -> hex -> html so this is not related to Script serialization/deserialization.

Noticeably that the shorter data that fits OP_PUSHDATA_1 below gets processed correctly:

content = '<!DOCTYPE html><html><body><img width=100 height=100 src="data:image/webp;base64,UklGRkAAAABXRUJQVlA4WAoAAAAgAAAABAAABAAAVlA4ICIAAABwAQCdASoFAAUAD8D+JaACdAFAAAD+h8wfGV3w7ubr0rAA"></body></html>'.encode("utf-8")

successful rawtx:

02000000000102f8769f44ecd5f5ec8b9734e300e1bc2facb81899558c262635b2fda975d6503e0000000000ffffffff89e75be2e530c2a9633a244dd42f799a891fcc3bf7a6de59474c2a22c41a80b20000000000ffffffff024605000000000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3df0ca052a01000000225120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d0140fb239e813a123689a85ded8769478c457b8a23affb698743f1ae142fb21b49d4cdeca37430f3e0f8c8ae753587a90631ac938ee79f3050c41a559cb11241e277034043c0f636ef50c2b4f791c1d2ef00be6a57097ffd93837817b3eed8a0e027532c613d6a36529af71b29196955fdb74ce568b060fb8c7c802a99723bfd63ffe36afd070120c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebcaac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004cc13c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f776562703b6261736536342c556b6c47526b41414141425852554a51566c413457416f4141414167414141414241414142414141566c413449434941414142774151436441536f46414155414438442b4a614143644146414141442b68387766475633773775627230724141223e3c2f626f64793e3c2f68746d6c3e6821c0c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebca00000000

decoded:

{'result': {'txid': 'dd4cea559846a523a0f7fd690c4851680b663e6a5c940ec29560417e918768e4',
            'hash': 'd73d7ae93b1a608bb5951d7c3340a5231bed37c756c0b39cddb1049fd8437400',
            'version': 2,
            'size': 612,
            'vsize': 287,
            'weight': 1146,
            'locktime': 0,
            'vin': [{'txid': '3e50d675a9fdb23526268c559918b8ac2fbce100e334978becf5d5ec449f76f8',
                     'vout': 0,
                     'scriptSig': {'asm': '', 'hex': ''},
                     'txinwitness': ['fb239e813a123689a85ded8769478c457b8a23affb698743f1ae142fb21b49d4cdeca37430f3e0f8c8ae753587a90631ac938ee79f3050c41a559cb11241e277'],
                     'sequence': 4294967295},
                    {'txid': 'b2801ac4222a4c4759dea6f73bcc1f899a792fd44d243a63a9c230e5e25be789',
                     'vout': 0,
                     'scriptSig': {'asm': '', 'hex': ''},
                     'txinwitness': ['43c0f636ef50c2b4f791c1d2ef00be6a57097ffd93837817b3eed8a0e027532c613d6a36529af71b29196955fdb74ce568b060fb8c7c802a99723bfd63ffe36a',
                                     '20c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebcaac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004cc13c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f776562703b6261736536342c556b6c47526b41414141425852554a51566c413457416f4141414167414141414241414142414141566c413449434941414142774151436441536f46414155414438442b4a614143644146414141442b68387766475633773775627230724141223e3c2f626f64793e3c2f68746d6c3e68',
                                     'c0c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebca'],
                     'sequence': 4294967295}],
            'vout': [{'value': 1.35e-05,
                      'n': 0,
                      'scriptPubKey': {'asm': '1 '
                                              'a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d',
                                       'desc': 'rawtr(a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d)#vm7gj0as',
                                       'hex': '5120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d',
                                       'address': 'bcrt1p5um09qqgggcxtkm30r6xf602qxw5f5u47j375jrstm9h06ffts7s6t55e6',
                                       'type': 'witness_v1_taproot'}},
                     {'value': 49.9999,
                      'n': 1,
                      'scriptPubKey': {'asm': '1 '
                                              'a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d',
                                       'desc': 'rawtr(a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d)#vm7gj0as',
                                       'hex': '5120a736f28008423065db7178f464e9ea019d44d395f4a3ea48705ecb77e9295c3d',
                                       'address': 'bcrt1p5um09qqgggcxtkm30r6xf602qxw5f5u47j375jrstm9h06ffts7s6t55e6',
                                       'type': 'witness_v1_taproot'}}]},
 'error': None,
 'id': 0}

witness script decoded:

{'result': {'asm': 'c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebca '
                   'OP_CHECKSIG 0 OP_IF 6582895 1 '
                   '746578742f68746d6c3b636861727365743d7574662d38 0 '
                   '3c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f776562703b6261736536342c556b6c47526b41414141425852554a51566c413457416f4141414167414141414241414142414141566c413449434941414142774151436441536f46414155414438442b4a614143644146414141442b68387766475633773775627230724141223e3c2f626f64793e3c2f68746d6c3e '
                   'OP_ENDIF',
            'desc': 'raw(20c29ab360da10dbcfe26000e13232911bcedc83bfd4c758ad7eaaed5f5ef8ebcaac0063036f7264010117746578742f68746d6c3b636861727365743d7574662d38004cc13c21444f43545950452068746d6c3e3c68746d6c3e3c626f64793e3c696d672077696474683d313030206865696768743d313030207372633d22646174613a696d6167652f776562703b6261736536342c556b6c47526b41414141425852554a51566c413457416f4141414167414141414241414142414141566c413449434941414142774151436441536f46414155414438442b4a614143644146414141442b68387766475633773775627230724141223e3c2f626f64793e3c2f68746d6c3e68)#qpm8k9t2',
            'type': 'nonstandard',
            'p2sh': '2N2ceYfT3HFgj1JvUTPZeaGG85KjdmCQdNQ',
            'segwit': {'asm': '0 '
                              '9f01eb3bbe63f4189c4aaf4a0061a21cb66d9a45365c8e912c07673aed33a19c',
                       'desc': 'addr(bcrt1qnuq7kwa7v06p38z24a9qqcdzrjmxmxj9xewgayfvqann4mfn5xwqtd4ljp)#7fpphmzh',
                       'hex': '00209f01eb3bbe63f4189c4aaf4a0061a21cb66d9a45365c8e912c07673aed33a19c',
                       'address': 'bcrt1qnuq7kwa7v06p38z24a9qqcdzrjmxmxj9xewgayfvqann4mfn5xwqtd4ljp',
                       'type': 'witness_v0_scripthash',
                       'p2sh-segwit': '2MvLxkLa183khKR4MUppXHqAyWQo7qA3E6X'}},
 'error': None,
 'id': 0}

All other code and datapath are absolutely same, plus I've isolated #68 using pickle library for serdes and additional assertions that guarantee consistency of taproot script hex. So it seems the issue is either OPCODE-specific (affects OP_PUSHDATA_2) or size-specific or content-specific.

ongrid commented 2 months ago

I've figured out the root cause of the mandatory-script-verify-flag-failed (Push value size limit exceeded) error. In Bitcoin Core, when processing the sendrawtransaction command, it requires that the length of the "token" in the script does not exceed MAX_SCRIPT_ELEMENT_SIZE = 520 https://github.com/bitcoin/bitcoin/blob/0de63b8b46eff5cda85b4950062703324ba65a80/src/script/script.h#L27

If you encounter this error, you'll need to break down long data into smaller tokens (I used 512 bytes for better kilobyte-alignment), and then use these minimized tokens in the script. Here’s a function and an example of how to use it, shown below.

# To avoid `mandatory-script-verify-flag-failed (Push value size limit exceeded)` in bitcoin-core
# See https://github.com/bitcoin/bitcoin/blob/0de63b8b46eff5cda85b4950062703324ba65a80/src/script/script.h#L27
# and https://github.com/karask/python-bitcoin-utils/issues/69
def chunk_script_element(data: bytes):

    BTC_CORE_MAX_SCRIPT_ELEMENT_SIZE = 512

    for i in range(0, len(data), BTC_CORE_MAX_SCRIPT_ELEMENT_SIZE):
        yield data[i:i+BTC_CORE_MAX_SCRIPT_ELEMENT_SIZE]

Example use for ordinals case

            taproot_script_p2pk = Script(
                [
                    priv_key.get_public_key().to_x_only_hex(),
                    "OP_CHECKSIG",
                    "OP_0",
                    "OP_IF",
                    "ord".encode("utf-8").hex(),
                    "01",
                    MIME_TYPE.encode("utf-8").hex(),
                    "OP_0",
                    *[chunk.hex() for chunk in chunk_script_element(content)],
                    "OP_ENDIF",
                ]
            )
ongrid commented 2 months ago

@karask maybe incorporate this chunking approach in Script class by default?

karask commented 2 months ago

Hi @ongrid and thanks for the feedback

The tx size block weight limit for a user (not miner) is 400kBs, not 500.

I went through the issues quite quickly just to get an idea of the problem. Indeed, data pushes need to be in chunks of 520 bytes, so for inscriptions you would need sth like this:

OP_FALSE OP_IF OP_PUSH "ord" OP_1 OP_PUSH "text/plain;charset=utf-8" OP_0 OP_PUSH "max 520 bytes data" OP_ENDIF ...rest of unlocking conditions...

I have not tested this limit myself and I will when I find some time. I was away in an event and I will soon be away again so not sure when I will spent some time on this.

Meanwhile make sure you try <400kBs txs when testing.

karask commented 2 months ago

@karask maybe incorporate this chunking approach in Script class by default?

Hi @ongrid ,

For now, the library rarely does execution validation; it leaves this to the node. Validating txs in the library would involve a ton of work (and maintenance to keep it in sync with Bitcoin core). Maybe a check could be added to see if the size is >520 but definitely not a priority.