Msgpacktool encode not working with array of bytes

[ori-shem-tov]

Subject of the issue

When decoding and re-encoding a transaction containing an array of bytes with msgpacktool, the re-encoded transaction is different from the original one. The issue seems to be that when decoding to JSON, arrays of bytes are encoded to arrays of base64 strings. But when encoding, these arrays of base64 strings are directly considered arrays of bytes.

Your environment

• Software version:

  8590000132
  2.1.4.stable [rel/stable] (commit #65e359d6)

• Node status if applicable:

  Last committed block: 9687902
  Time since last block: 3.4s
  Sync Time: 0.0s
  Last consensus protocol: https://github.com/algorandfoundation/specs/tree/3a83c4c743f8b17adfd73944b4319c25722a6782
  Next consensus protocol: https://github.com/algorandfoundation/specs/tree/3a83c4c743f8b17adfd73944b4319c25722a6782
  Round for next consensus protocol: 9687903
  Next consensus protocol supported: true
  Last Catchpoint:
  Genesis ID: testnet-v1.0
  Genesis hash: SGO1GKSzyE7IEPItTxCByw9x8FmnrCDexi9/cOUJOiI=

• Operating System details: macOS Catalina 10.15.6

Steps to reproduce

Here is an example where this creates an issue:

1. Create an Algorand account (U2XBF6OUUUDOU6PTPZTEXUONIWLXEXTNHIGCMQ5YADQVXWCCMSDGI4WLTU)
2. Create the following simple TEAL script (replace the address with your account):

   #pragma version 2
   arg 0
   arg 1
   addr U2XBF6OUUUDOU6PTPZTEXUONIWLXEXTNHIGCMQ5YADQVXWCCMSDGI4WLTU
   ed25519verify
   int 1
   ==

3. Compile the above TEAL:

   goal clerk compile simple_script.teal -o lsig -a U2XBF6OUUUDOU6PTPZTEXUONIWLXEXTNHIGCMQ5YADQVXWCCMSDGI4WLTU -s

4. Create a transaction signed with the above lsig. Pass some base64 bytes as the first arg:

   goal clerk send -f U2XBF6OUUUDOU6PTPZTEXUONIWLXEXTNHIGCMQ5YADQVXWCCMSDGI4WLTU -t U2XBF6OUUUDOU6PTPZTEXUONIWLXEXTNHIGCMQ5YADQVXWCCMSDGI4WLTU -a 1000 -o txn -L lsig --argb64 MTIzCg==

5. Create a keyfile for your account:

   algokey import -m "arrive draw near half hedgehog width fluid monster corn neglect milk egg chimney team tank hurdle grape hip either glow gadget salmon donor absent eyebrow" -f keyfile

6. Sign the base64 bytes with your account and add it to the transaction as a second arg:

   goal clerk tealsign --lsig-txn txn --data-b64 MTIzCg== --set-lsig-arg-idx 1 --keyfile keyfile

7. At this point you should be able to run goal clerk dryrun -t txn on the transaction and it should pass.
8. Decode the transaction with msgpacktool -d -b32 < txn > txn.decoded
9. Encode the the above output with msgpacktool -e -b32 < txn.decoded > txn2
10. Run goal clerk dryrun -t txn2

Expected behaviour

dryrun should pass

Actual behaviour

dryrun fails

[yaovi-a]

Need to better understand what needs to be done here.

[fabrice102]

The issue seems to be that, when encoding to JSON, if a field is an array of bytes [][]byte, then the field is converted, in JSON to an array of base64 strings but the name of the field is not changed. Then, when decoding from JSON, the array of strings is interpreted as an array of string []string, instead of decoding it into an array of bytes.

I think the best solution may be to use the same trick as for encoding bytes ([]byte): change the name of the field to xxx:b64.

Note that ideally, this should work for arrays of arrays of bytes too and so on ([][][]byte, [][][][]byte, ....).