Can't submit transaction with raw json datum

[grananqvist]

I am trying to make a transaction with datum attached to output (not spend a plutus UTxO, but like making a regular transaction with --tx-out-datum-embed-file from cardano-cli). This is required when submitting to many of the existing smart contracts on Cardano. This is my attempt:

network = pc.Network.TESTNET 
psk = pc.PaymentExtendedSigningKey.load("testnet-keys/00.skey")
ssk = pc.StakeExtendedSigningKey.load('testnet-keys/stake.skey')
pvk = pc.PaymentExtendedVerificationKey.from_signing_key(psk)
svk = pc.StakeExtendedVerificationKey.from_signing_key(ssk)
address = pc.Address(pvk.hash(), svk.hash(), network)
context = pc.BlockFrostChainContext(my_token, network)
builder = pc.TransactionBuilder(context)

builder.add_input_address(address)

datum = {b'fields': [], b'constructor': 0}
datum_hash = pc.DatumHash(blake2b(cbor2.dumps(datum, default=default_encoder), 32, encoder=RawEncoder))

builder.add_output(pc.TransactionOutput(pc.Address.from_primitive("addr_test1vrm9x2zsux7va6w892g38tvchnzahvcd9tykqf3ygnmwtaqyfg52x"),
                         amount=pc.Value(10000000), datum_hash=datum_hash))

tx_body = builder.build(change_address=address)
signature = psk.sign(tx_body.hash())
vk_witnesses = [pc.VerificationKeyWitness(pvk, signature)]
tx = pc.Transaction(tx_body, pc.TransactionWitnessSet(vkey_witnesses=vk_witnesses,
                           plutus_data=[datum]))

context.submit_tx(tx.to_cbor())

It results in a node error on submission saying the fees were miscalculated:

ApiError: {'error': 'Bad Request', 'message': '"transaction submit error ShelleyTxValidationError ShelleyBasedEraAlonzo (ApplyTxError [UtxowFailure (PPViewHashesDontMatch SNothing (SJust (SafeHash \\"7e58e4a25bc56a14475e7461cda5aeeaf59fff97285560887a9eedd2ddea1d9f\\"))),UtxowFailure (WrappedShelleyEraFailure (UtxoFailure (FeeTooSmallUTxO (Coin 169241) (Coin 168317))))])"', 'status_code': 400}

I can temporarily fix this by making sure datum is included when calculating fee (this will need a proper fix):

class MyTransactionBuilder(pc.TransactionBuilder):     
    def _build_fake_witness_set(self) -> pc.TransactionWitnessSet:
        return pc.TransactionWitnessSet(
            vkey_witnesses=self._build_fake_vkey_witnesses(),
            native_scripts=self.native_scripts,
            plutus_data=self.plutus_data
        )
# ... add stuff to builder
builder.plutus_data = [datum]
# .. build tx and submit

Then I am left with this error:

ApiError: {'error': 'Bad Request', 'message': '"transaction submit error ShelleyTxValidationError ShelleyBasedEraAlonzo (ApplyTxError [UtxowFailure (PPViewHashesDontMatch SNothing (SJust (SafeHash \\"7e58e4a25bc56a14475e7461cda5aeeaf59fff97285560887a9eedd2ddea1d9f\\")))])"', 'status_code': 400}

A quick google search indicate that this might have something to do with the cost model not being in protocol params. It doesn't look like the protocol params are used anywhere when creating the transaction with TransactionBuilder ? I know Plutus is not all supported yet in PyCardano, but I thought adding a raw json datum to the transaction would be different.

• PyCardano Version 0.2.0

  Thanks.

[cffls]

Hi @grananqvist,

Thanks for reporting this issue. After digging a bit, I found that the problem is that we need to also attach script_data_hash to transaction body when there is datum or redeemer attached.

From your example, I changed tx builder to the following:

class MyTransactionBuilder(pc.TransactionBuilder):
    def _build_fake_witness_set(self) -> pc.TransactionWitnessSet:
        return pc.TransactionWitnessSet(
            vkey_witnesses=self._build_fake_vkey_witnesses(),
            native_scripts=self.native_scripts,
            plutus_data=self.plutus_data
        )

    def _build_tx_body(self) -> pc.TransactionBody:
        tx_body = pc.TransactionBody(
            [i.input for i in self.inputs],
            self.outputs,
            fee=self.fee,
            ttl=self.ttl,
            mint=self.mint,
            auxiliary_data_hash=self.auxiliary_data.hash()
            if self.auxiliary_data
            else None,
            script_data_hash=self.script_data_hash,
            required_signers=self.required_signers,
            validity_start=self.validity_start,
        )
        return tx_body

and added this line before building the final transaction body:

builder.script_data_hash = pc.script_data_hash([], [datum])

The transaction was submitted successfully after these changes. Here is the transaction record: https://testnet.cardanoscan.io/transaction/9b5382ca13046a5ca3a5d5c37abc384b4c59a7ceceb1a257e1b1f4cbcea4b7eb

The datum was attached to the 1.5ADA output. Its hash is 64426970d9fd15124a2f214469a6b1ba1db907c271d04364d43605f41da48dfc, and could be successfully queried from Blockfrost:

curl -s -H "Content-Type: application/json" -H "project_id: my_testnet_project_id" https://cardano-testnet.blockfrost.io/api/v0/scripts/datum/64426970d9fd15124a2f214469a6b1ba1db907c271d04364d43605f41da48dfc | jq
{
  "json_value": {
    "map": [
      {
        "k": {
          "bytes": "6669656c6473"
        },
        "v": {
          "list": []
        }
      },
      {
        "k": {
          "bytes": "636f6e7374727563746f72"
        },
        "v": {
          "int": 0
        }
      }
    ]
  }
}

Notice that the returned json is not exactly the same as the one submitted before. My guess is that the datum is reformatted by cardano-db-sync.

PlutusData can restore the json to a form that is very close to its original format:

pc.PlutusData.from_dict(block_frost_datum["json_value"])
{b'fields': <pycardano.serialization.IndefiniteList object at 0x102137be0>, b'constructor': 0}

The only different part is that the original list became an IndefiniteList (due to a special CBOR serialization requirement), but you can retrieve the actual list by calling its field items.

[grananqvist]

Great! That makes sense. However, the datum returned from blockfrost in this case is very weird indeed. Querying for another random datum returns expected format:

curl -s -H "Content-Type: application/json" -H "project_id: my_id" https://cardano-testnet.blockfrost.io/api/v0/scripts/datum/40e9733adc2315254fb33ba8e1eea60c4269b65e3c1ff3c4790750214ce11ed6
{"json_value":{"fields":[{"bytes":"5301"},{"fields":[{"fields":[{"fields":[{"fields":[{"bytes":"b838a022a689bc05dc05fad36b3681e83c7c5d83f8c2d31a0aaf966e"}],"constructor":0},{"fields":[{"fields":[{"fields":[{"bytes":"34c7120baec5a47b25384006529986a761f71f6902d65c22b1a18d6b"}],"constructor":0}],"constructor":0}],"constructor":0}],"constructor":0},{"fields":[],"constructor":1}],"constructor":0},{"fields":[],"constructor":1}],"constructor":0},{"int":2500000},{"fields":[{"fields":[],"constructor":0},{"int":28078251},{"fields":[{"int":9000000000}],"constructor":0}],"constructor":0}],"constructor":0}}

[cffls]

This format is specially designed for Plutus, so the datum could be correctly reconstructed when passed to cardano-cli. You can find the Haskell function that formats the json here: https://github.com/input-output-hk/cardano-node/blob/baa9b5e59c5d448d475f94cc88a31a5857c2bda5/cardano-api/src/Cardano/Api/ScriptData.hs#L449-L474

The function is used by cardano-db-sync to transform the cbor whenever it sees a datum. I agree that it does not makes sense to format raw json in this case, because it is never meant to be used in Plutus script. Maybe this behavior should be corrected in db-sync. For now, you can use the from_dict function mentioned above to temporarily get around this issue.

[grananqvist]

Ok, so you are saying this is the intended behaviour?

When I change the datum from raw json to type pc.PlutusData:

from dataclasses import dataclass
@dataclass
class A(pc.PlutusData):
    CONSTR_ID = 0
datum = A()

datum_hash = pc.DatumHash(blake2b(cbor2.dumps(datum, default=default_encoder), 32, encoder=RawEncoder))
builder.plutus_data = [datum]
builder.script_data_hash = pc.script_data_hash([], [datum])

builder.add_output(pc.TransactionOutput(address,
                         amount=pc.Value(11000000), datum_hash=datum_hash))

then I get what I expect from inspecting the datum on chain with blockfrost:

 curl -s -H "Content-Type: application/json" -H "project_id: my_id" https://cardano-testnet.blockfrost.io/api/v0/scripts/datum/923918e403bf43c34b4ef6b48eb2ee04babed17320d8d1b9ff9ad086e86f44ec
{"json_value":{"fields":[],"constructor":0}}

[cffls]

Didn't mean it is the intended behavior. Instead, I meant it is an expected behavior due to the fact that db-sync will always reformat the datum regardless of its purpose, either raw json (your first example) or Plutus datum (your second example). The second example you posted has a more concise format returned by db-sync because your datum is encoded through PlutusData, which follows the CBOR encoding rules for Plutus datum, so it could be correctly interpreted by db-sync. In the end, it depends on how you want to use the datum after. If you want to pass it to Plutus script, then it is better to inherit your datum from PlutusData.

[grananqvist]

Ok, thanks for clarification, I will stick to PlutusData then

[cffls]

Closing this issue as the question was answered. Also, in the latest version of tx builder, script_data_hash will be automatically included in a transaction body output.