Make JSON encoding total & standardize it

[klntsky]

Currently, not every transaction can be converted to JSON. See #634 and its cause. This is not reflected in the type of to_json method.

The context of the issue is: I am pushing forward an initiative to standardize JSON representations of Cardano domain types - see this CPS for info on motivation.

In order to build a universal schema, we need a "source of truth", and preferably we would like to use one of the existing implementations. CSL looks like the best candidate, because almost every offchain library is using it already. But a non-total to_json is not an option.

I'm willing to contribute to this project. Here's what I can do:

• Identify all sources of partiality in the JSON encoding/decoding code
• Propose encoding adjustments
• Implement them in code and make a PR

After that, I would produce a schema and push it through the CIP acceptance process.

Would you agree to "freeze" the JSON encoding implementations after that and only evolve them in sync with the standard (CIP)?

[klntsky]

In addition, if you agree to a breaking change, I think it would make sense to synchronize CSL with cardano-cli JSON format as much as possible.

[klntsky]

For the specific issue with PlutusData, I think the way forward would be to introduce a new PlutusDatumSchema that handles arbitrary-length integers, and make its use the default (i.e. use it for JSON-encoding of any types that contain PlutusData)

[rooooooooob]

One issue here is how it's all auto implemented using macros. If anything changes in the serde/jsonschema crates it'd change how it works. It also means a lot of the json is quite ugly. Metadata follows Cardano node for example but the rest doesn't. I think Plutus datum might or maybe it's just CML. The other 95%+ is auto generated.

[rooooooooob]

For the integers it's a restriction in the json library CSL uses. It would have to be fixed there to allow outside the i64 range.

[klntsky]

For the integers it's a restriction in the json library CSL uses. It would have to be fixed there to allow outside the i64 range.

Yep we know how painful dealing with long integers in JSON can be. Forking any of the library tooling just for that is probably too expensive.

[lisicky]

Thanks @rooooooooob for your answer!

@klntsky standardization is a good thing, but now CSL has autogenerated schemas and default JSON de/serializators for most of the protocol types and it requires not small amount of time to switch from default serde/jsonschema implementations for each type to self-written schemas and serialization/deserialization logic. But I guess it's better to start from creating CIP with JSON schema proposal for each protocol type. Because if you spent time to create a PR with big bunch of code and your CIP would not be accepted it might be wasting of your time. For values bigger than i64 there is one workaround that I didn't test yet, but I will try to check it before CSL 12 will be released. Anyway feel free to ask questions, we might help if you stuck with code or protocol analysis.

CC @vsubhuman

[klntsky]

@lisicky here's a CIP draft: https://github.com/cardano-foundation/CIPs/pull/766/

I tried to synchronize it with CSL naming of types where possible

[klntsky]

The conventions for naming/encoding data there are different from those used in CSL - could someone please take a look and share an opinion on them

[lisicky]

@klntsky It was fast : ) I'll take a look during the next week.

[klntsky]

This is a compatibility table between JSON representation layouts in CSL and the cardano json-schema CIP.

---

[x] means that the layout of a type in CSL is compatible with the JSON-schema (but not necessarily all the structural sub-components of the type! This just indicates whether there are changes needed in code for the type itself) [ ] means that there are some differences (that are listed below)

---

• [x] Address - uses bech32
• [ ] AuxiliaryData
  • does not include prefer_alonzo_format (CSL-specific)
• [x] AuxiliaryDataHash
• [x] AuxiliaryDataSet
• [ ] BaseAddress
  • CSL does not offer to_json
• [ ] BigInt
  • CSL serializes as JSON number when used in PlutusData:

    > lib.PlutusData.new_integer(lib.BigInt.from_str('1239999999999999999999999999999999999999999')).to_json()
    Uncaught 'Integer 1239999999999999999999999999999999999999999 too big for our JSON support'
    > lib.BigInt.from_str('9999999999999999999').to_json()
    '"9999999999999999999"'

• [x] Block
• [x] BlockHash
• [ ] BootstrapWitness
  • attributes is an array of numbers in CSL instead of hex(bytes)
  • chain_code is an array of numbers in CSL instead of hex(bytes)
• [ ] ByronAddress
  • CSL does not offer to_json
• [x] ByteString - both use hex encoding
• [ ] Certificate
  • does not use tag property
• [x] CostModels
• [ ] Credential
  • does not use tag property
• [x] DNSName - DNSRecordSRV in CSL
• [x] DataHash
• [x] Ed25519KeyHash
• [x] Ed25519PublicKey
• [x] Ed25519Signature
• [ ] EnterpriseAddress
  • CSL does not offer to_json
• [x] ExUnitPrices
• [x] ExUnits
• [x] GenesisDelegateHash
• [x] GenesisHash
• [x] Header
• [x] HeaderBody
• [ ] HeaderLeaderCert
  • does not use tag property
• [x] Int128
• [ ] Ipv4
  • CSL represents as an array of 4 bytes, we represent as a string
• [ ] Ipv6
  • CSL represents as an array of 12 bytes, we represent as a string
• [x] KESSignature - only used in HeaderJSON.body_signature
• [x] KESVKey
• [ ] Language
  • CIP uses snake_case for enums
• [ ] MIRPot
  • CIP uses snake_case for enums
• [ ] Mint
  • incompatible
• [ ] MoveInstantaneousRewards
  • does not use tag property, we don't use MIREnumJSON
• [x] MultiAsset
• [ ] NativeScript
  • does not use tag property
• [ ] NetworkId
  • CIP uses snake_case for enums
• [ ] Nonce
  • not compatible
• [x] OperationalCert
• [ ] PlutusData
  • does not use tag property
• [ ] PlutusScript
  • does not specify language field, encodes as binary
• [x] PlutusV1CostModel
• [x] PlutusV2CostModel
• [ ] PointerAddress
  • CSL does not offer to_json
• [x] PoolMetadata
• [x] PoolParams
• [x] ProtocolParamUpdate
• [x] ProtocolVersion
• [x] Redeemer
• [ ] RedeemerTag
  • CIP uses snake_case for enums
• [ ] Relay
  • does not use tag property
• [ ] RewardAddress
  • CSL does not offer to_json
• [x] ScriptDataHash
• [x] ScriptHash
• [ ] ScriptRef
  • does not use tag property
• [x] Transaction
• [x] TransactionBody
• [x] TransactionHash
• [x] TransactionInput
• [ ] TransactionMetadata
  • incompatible with GeneralTransactionMetadata: uses key/value pairs
• [ ] TransactionMetadatum
  • does not use tag property
• [ ] TransactionOutput
  • plutus_data uses different format (tagged)
• [x] TransactionUnspentOutput
• [ ] TransactionWitnessSet
  • rename vkeys field to vkeywitnesses
• [x] UInt32 - represented as number, does not overflow
• [x] UInt64
• [x] URL
• [x] UnitInterval
• [x] Update
• [ ] VRFCert
  • output and proof fields must be hexstrings
• [x] VRFKeyHash
• [x] VRFVKey
• [x] Value
• [x] Vkeywitness

[lisicky]

@klntsky now it should work in the lates CSL alpha lib.PlutusData.new_integer(lib.BigInt .... big int