(0.96.5) Offline Transaction created with segwit related input not decodable/recognizable by offline (or original) client

[achamely]

@goatpig We run an online web wallet (Omniwallet) that offers armory offline transaction support. Randomly we have noticed that some users are getting offline transactions that appears to not be decodable/signable by their offline clients. We have confirmed our local library used by the web wallet is 0.96.5 and the clients are using 0.96.5 as well.

We are using the UnsignedTransaction().fromJSONMap(json_nosig, True).serializeAscii() function to create the unsigned tx (https://github.com/OmniLayer/omniwallet/blob/master/api/armory_service.py#L91)

During our investigations we have found that the issue seems to occur when one of the utxos involved in the offline tx had a segwit input or output. Digging into the code and doing testing it appears the problem might be in how the supporttx is being encoded in the fromJSONMap function as when doing our testing once encoded it is no longer decodable.

Here is an example of a rawtx and the related armory encoded tx which demonstrates the issue. rawtx: 0100000001efd993ea0c6db36d3e86e151042be1bc68593c1b48b1470b76e662d47a480ec9000000001976a914f10c2aaff99ac010c293a4a67239b9d52d71b16288acffffffff0318220000000000001976a914f10c2aaff99ac010c293a4a67239b9d52d71b16288ac22020000000000001976a914c03801a7c1e118b0eb8532c4dfa04a5e8779a51b88ac0000000000000000166a146f6d6e690000000000000003000000000000000a00000000

The utxo used by this tx (which contains a segwit input itself) is https://www.blockchain.com/btc/tx/c90e487ad462e6760b47b1481b3c5968bce12b0451e1863e6db36d0cea93d9ef

Once we pass this information to our armory service we get the following armory unsigned tx

=====TXSIGCOLLECT-Cwztf8uP======================================
AQAAAPm+tNkAAAAAAf1bAQEAAAD5vrTZ79mT6gxts20+huFRBCvhvGhZPBtIsUcL
duZi1HpIDskAAAAA4gIAAAAAAQEUTbsjlSzpC1SnlMIz8e2YwL2EtnwfpwEIqUxs
l3tBHAAAAAAA/v///wIQJwAAAAAAABl2qRTxDCqv+ZrAEMKTpKZyObnVLXGxYois
RS4BAAAAAAAWABRGeQNiYQ6oIXf8f5WgOR1mtw0dMAJIMEUCIQDgqkn9ZqAZbGOS
oxRtHI61Tf3PofFx3/bH/MVuRhl5BAIgK58AaKqVaPSPglbFqtoNOuOgVSYM10wv
PoOBgn/qilEBIQJ69CEBbSAEXYByJ96DZMN2gfbd/D1gprXdKszLce6ptVASCQAA
AAD/////AUEErUSNPb/lD6ROzNNr7PZgNgjVwqvNh4rrRU49o5Z6mRCyORKYjuDR
7yWQmM/41X7d/WqYTwGHpEfm7A66NGsgxQAAAzQBAAAA+b602Rl2qRTxDCqv+ZrA
EMKTpKZyObnVLXGxYoisGCIAAAAAAAAAAAROT05FAAAANAEAAAD5vrTZGXapFMA4
AafB4Riw64UyxN+gSl6HeaUbiKwiAgAAAAAAAAAABE5PTkUAAAAxAQAAAPm+tNkW
ahRvbW5pAAAAAAAAAAMAAAAAAAAACgAAAAAAAAAAAAAETk9ORQAAAAAAAAAgB0Rl
ZmF1bHQwqgEAAAAAAAAAEAAAAAABNAAB/////wYk79mT6gxts20+huFRBCvhvGhZ
PBtIsUcLduZi1HpIDskAAAAAECcAAAAAAAADIhgiAAAAAAAAGXapFPEMKq/5msAQ
wpOkpnI5udUtcbFiiKwiIgIAAAAAAAAZdqkUwDgBp8HhGLDrhTLE36BKXod5pRuI
rB8AAAAAAAAAABZqFG9tbmkAAAAAAAAAAwAAAAAAAAAK
================================================================

However this unsigned tx is no longer decodable by the clients library or our own, which just created it.

Thoughts?

[goatpig]

Transaction created by 0.96.5 have to be signed by an offline 0.96.5 instance. You can look at the commit history for 0.96.4 & 0.96.5, it's quite easy to spot the changes.

This was my only solution to offer support for bech32 and nested SW, which nests twice. The existing python code hardcoded the p2sh container processing.

This approach allowed for legacy transactions to be signed by older armory, but naturally signing SW inputs had to fail.

[achamely]

@goatpig that's the issue. Both the online and offline client are 0.96.5 However the offline 0.96.5 does not seem to be able to decode/load the offline tx that was created with the online 0.96.5 library

[goatpig]

I see you're using the JSON codec. I don't remember updating that part to support SW. The JSON stuff is for armoryd, which hasn't received the SW port.

[achamely]

Thanks @goatpig that confirms that the function we are using needs to be changed. We will set some time up to look into what updates are necessary. Can you confirm/give us a pointer at which functions for the UnsignedTransaction creation support segwit inputs that we should be using?

[goatpig]

It seems that at some point the JSON stuff was fixed. This commit in particular is of interest: f73c2fbe4a

In general, these are the commits that changed the UnsignedTransaction serialization for nested SW support:

7f2a537e f73c2fbe 1f847554 7a479f9b

armoryengine/Transaction.py is where it all takes place for each commit.

[achamely]

@goatpig i've been looking through the commits and trying a few different methods but still seem to be hitting some blockers.

First i tried to use UnsignedTransaction().createFromUnsignedTxIO by passing in an array of objects created with UnsignedTxInput().fromJSONMap and DecoratedTxOut().fromJSONMap . While this produced an AsciiSerialized Armory offline object it seemed to have the same issue as originally mentioned with not being recognized.

After that i tried to use UnsignedTransaction().createFromPyTx and just straight UnsignedTransaction() I used PyTx().unserialize(hex_to_binary(unsigned_raw_hex)) to get the initial pytx object and then i constructed the txmap and pubkeymap objects to pass through as well using the following code snippet (for reference i start with a fully created unsigned transaction in rawhex and the users pubkey string for the sending address)

pubKeyMap={}
pubKeyMap[SCRADDR_P2PKH_BYTE+hash160(pubkey)]= pubkey
pytx=PyTx().unserialize(hex_to_binary(unsigned_hex))
decoded_tx = decoderawtransaction(unsigned_hex)['result']
txMap ={}
for intx in decoded_tx['vin']:
  spending_txid= intx['txid']
  i_vout = intx['vout']
  spending_tx_raw = getrawtransaction(spending_txid)['result']['hex']
  support_tx=PyTx().unserialize(hex_to_binary(spending_tx_raw))
  txMap[support_tx.thisHash]=support_tx
  txout = support_tx.outputs[i_vout]
  scrAddr = script_to_scrAddr(txout.getScript())
  pubKeyMap[scrAddr]=pubkey

UnsignedTransaction(pytx,pubKeyMap,txMap).serializeAscii()

At first I ran into some issues with just passing in the straight pubkey as a string for pubKeyMap as it appears in two different places that it was trying to look up the same info in 2 different ways (first as the SCRADDR_P2PKH_BYTE+hash160(pubkey) then as script_to_scrAddr(txout.getScript()) so i had to add the code above to construct both elements. While this does seem to generate an Armory Ascii Serialized object that handles the segwit utxo properly and that the client can now at least identify, it still seems to be unable to decode / sign it. Trying to unserialize it locally i get an error related to a missing pubkey map for singlesig USTXI so i'm not sure what i missed as this should have been included in the ascii string that unserialize uses to reconstruct the object when called.

Any pointers or suggestions you can provide would be appreciated.

[goatpig]

I have not touched this code in a while so I can only refer to distant memories to help you here. The python signer does not handle anything SegWit. The cpp signer does. The cpp signer has its own state that gets ser/deser along. You don't seem to be passing that at all (which is what is requires the hash to pubkey resolving).

https://github.com/goatpig/BitcoinArmory/blob/master/armoryengine/Transaction.py#L915

I'd be looking at that (the signer object in PyTx). Short of this, I can't really help you unless you write test code illustrating the entire process from unsigned tx serialization in your client to its deseriliazation and signing in the offline Armory instance.

[achamely]

@goatpig thanks, i'll take a look at the signer functions linked and dig around to see what i can pull out and apply.
As a side reference, we do not handle any of the deser/signing, we actually leave that entirely for your Armory client itself as installed/run by the user. The local deser we were doing was simply as a test to see if the serialized ascii would deserialize. Not sure what, if any, that might change your answer.
If we continue to have issues i will paste a full python test code with example info

[goatpig]

As a side reference, we do not handle any of the deser/signing, we actually leave that entirely for your Armory client itself as installed/run by the user.

I get your point but unless you isolate that code in a headless sample I'll have to rely on shenanigans on my end to get production Armory to even get started with a serialized unsigned tx you'd send.

It's just simpler for me if you import the Armory libs into the sample code so that I can watch entire stack running in the debugger.

[achamely]

Ok i've got a stand alone snippet you can run on your side to see how everything is generated. To simplify the process i've included the info that a bitcoin-core client would provide as well.

from armoryengine.ALL import *

#provided by the user
unsigned_hex = "0100000001efd993ea0c6db36d3e86e151042be1bc68593c1b48b1470b76e662d47a480ec9000000001976a914f10c2aaff99ac010c293a4a67239b9d52d71b16288acffffffff0318220000000000001976a914f10c2aaff99ac010c293a4a67239b9d52d71b16288ac22020000000000001976a914c03801a7c1e118b0eb8532c4dfa04a5e8779a51b88ac0000000000000000166a146f6d6e690000000000000003000000000000000a00000000"
pubkey = "04AD448D3DBFE50FA44ECCD36BECF6603608D5C2ABCD878AEB454E3DA3967A9910B23912988EE0D1EF259098CFF8D57EDDFD6A984F0187A447E6EC0EBA346B20C5"

#provided to simplify running code below
decoded_tx = {u'hash': u'c9b8956d942718fbf92a172b695dfea6c01264caf74f1306deb97808eda92087', u'vout': [{u'scriptPubKey': {u'reqSigs': 1, u'hex': u'76a914f10c2aaff99ac010c293a4a67239b9d52d71b16288ac', u'addresses': [u'1NyYRbK5BbvMmzT7uE8C7BNgCEbWNWMxUk'], u'asm': u'OP_DUP OP_HASH160 f10c2aaff99ac010c293a4a67239b9d52d71b162 OP_EQUALVERIFY OP_CHECKSIG', u'type': u'pubkeyhash'}, u'value': 8.728e-05, u'n': 0}, {u'scriptPubKey': {u'reqSigs': 1, u'hex': u'76a914c03801a7c1e118b0eb8532c4dfa04a5e8779a51b88ac', u'addresses': [u'1JXMqEH6B4zrNM1fBqhknLAYfj7cn8hKhw'], u'asm': u'OP_DUP OP_HASH160 c03801a7c1e118b0eb8532c4dfa04a5e8779a51b OP_EQUALVERIFY OP_CHECKSIG', u'type': u'pubkeyhash'}, u'value': 5.46e-06, u'n': 1}, {u'scriptPubKey': {u'type': u'nulldata', u'hex': u'6a146f6d6e690000000000000003000000000000000a', u'asm': u'OP_RETURN 6f6d6e690000000000000003000000000000000a'}, u'value': 0.0, u'n': 2}], u'vin': [{u'sequence': 4294967295, u'scriptSig': {u'hex': u'76a914f10c2aaff99ac010c293a4a67239b9d52d71b16288ac', u'asm': u'OP_DUP OP_HASH160 f10c2aaff99ac010c293a4a67239b9d52d71b162 OP_EQUALVERIFY OP_CHECKSIG'}, u'vout': 0, u'txid': u'c90e487ad462e6760b47b1481b3c5968bce12b0451e1863e6db36d0cea93d9ef'}], u'txid': u'c9b8956d942718fbf92a172b695dfea6c01264caf74f1306deb97808eda92087', u'version': 1, u'locktime': 0, u'vsize': 175, u'size': 175}
spending_tx_raw="02000000000101144dbb23952ce90b54a794c233f1ed98c0bd84b67c1fa70108a94c6c977b411c0000000000feffffff0210270000000000001976a914f10c2aaff99ac010c293a4a67239b9d52d71b16288ac452e01000000000016001446790362610ea82177fc7f95a0391d66b70d1d3002483045022100e0aa49fd66a0196c6392a3146d1c8eb54dfdcfa1f171dff6c7fcc56e4619790402202b9f0068aa9568f48f8256c5aada0d3ae3a055260cd74c2f3e8381827fea8a510121027af421016d20045d807227de8364c37681f6ddfc3d60a6b5dd2acccb71eea9b550120900"

#pubKeyMap=[pubkey] 
pubKeyMap={}
pubKeyMap[SCRADDR_P2PKH_BYTE+hash160(pubkey)]= pubkey

pytx=PyTx().unserialize(hex_to_binary(unsigned_hex))
#pulled from above
#decoded_tx = decoderawtransaction(unsigned_hex)['result']
txMap ={}
for intx in decoded_tx['vin']:
  spending_txid= intx['txid']
  i_vout = intx['vout']
  #pulled from above
  #spending_tx_raw = getrawtransaction(spending_txid)['result']['hex']
  support_tx=PyTx().unserialize(hex_to_binary(spending_tx_raw))
  support_tx.computeSignerState()
  txMap[support_tx.thisHash]=support_tx
  txout = support_tx.outputs[i_vout]
  scrAddr = script_to_scrAddr(txout.getScript())
  pubKeyMap[scrAddr]=pubkey

pytx.computeSignerState()

#z=UnsignedTransaction().createFromPyTx(pytx,pubKeyMap,txMap)
z=UnsignedTransaction(pytx,pubKeyMap,txMap)
z.serializeAscii()

[goatpig]

I'm busy this week, I can't address this just yet.

[achamely]

No worries. I'll continue to dig through. Any assistance you can provide later on when you get the chance would be appreciated