Allow XDR types to generate from / output to JSON

[morleyzhi]

Broadly, this allows a developer to compose XDR representations in JSON and convert them into real XDR objects (and vice versa).

The developer story here is: a developer reads the XDR definition, writes the transaction / asset / whatever in JSON, then is able to convert it to an XDR.

For example, given the following XDR definition of Transaction:

struct Transaction
{
    // account used to run the transaction
    AccountID sourceAccount;

    // the fee the sourceAccount will pay
    uint32 fee;

    // sequence number to consume in the account
    SequenceNumber seqNum;

    // validity range (inclusive) for the last ledger close time
    TimeBounds* timeBounds;

    Memo memo;

    Operation operations<100>;

    // reserved for future use
    union switch (int v)
    {
    case 0:
        void;
    }
    ext;
};

The JSON would look like:

{
    "sourceAccount": "G...",
    "fee": 100,
    "seqNum": 1234,
    "timeBounds": null,
    "memo": {
      "type": MEMO_NONE
    },
    "operations": [
      {
        "sourceAccount": null,
        "type": CREATE_ACCOUNT,
        "destination": "G...",
        "startingBalance": 200000000
      }
    ]
    "ext": 0 // maybe default to zero-value when omitted?
};

This example is for Transaction object but in general you should be able to do the same thing with any XDR object, you check it's definition, write JSON and you can quickly convert it to actual XDR representation.

There are some things to figure out:

• How do you represent XDR union, ex:

  union Memo switch (MemoType type)
  {
  case MEMO_NONE:
      void;
  case MEMO_TEXT:
      string text<28>;
  case MEMO_ID:
      uint64 id;
  case MEMO_HASH:
      Hash hash; // the hash of what to pull from the content server
  case MEMO_RETURN:
      Hash retHash; // the hash of the tx you are rejecting
  };

  An obvious way is to use union switch name type. So this would become:

  memo = {
    type: MEMO_TEXT,
    text: "Hello world!"
  }

  But maybe there's a better way? SEP-0011 is responsible for very similar thing but is using a different format than JSON so you can check it out.

• Some types are just bytes and we need to be able to pass a human-readable representation if there is any. For example, AccountID type (sourceAccount in Transaction) is PublicKey which is really:

  union PublicKey switch (PublicKeyType type)
  {
  case PUBLIC_KEY_TYPE_ED25519:
      uint256 ed25519;
  };

  Would be great to have a helper methods to quickly create byte arrays, ex:

  sourceAccount: AccountId("G..."),

• Other types require some modifications from human-readable form (ex. all amounts). We should probably have helper functions for this as well.

Originally posted by @bartekn in https://github.com/stellar/js-stellar-base/pull/157#issuecomment-462704769

[tomerweller]

The developer story here is: a developer reads the XDR definition, writes the transaction / asset / whatever in JSON, then is able to convert it to an XDR.

@morleyzhi Can you unpack this story a bit? Where is this JSON generated? Is it transmitted on a wire and then needs to turn into an XDR? Is the developer directly editing this JSON?

While JSON is clearly more human readable then XDR, it's still usually generated and consumed by computer programs. Perhaps instead of introducing a new JSON<->XDR mapping we can improve the generated XDR javascript classes by adding the type of accessors and mutators that are missing in our stories.

If it is meant for humans to interact with, maybe we can use TxREP (SEP11) as is? I'm not a huge fan of the grammar (would prefer using something like YAML) but it's better to have just one standard mapping.

[morleyzhi]

@bartekn Can you expand what the use case for this feature is? I haven't seen the exact community request that you're relaying.

FWIW as I've said in other tickets, I'm leaning towards not introducing more than one way to solve a problem, so if it's just a matter of giving developers a different way of assembling transactions, I'd opt to not do it.

[bartekn]

While JSON is clearly more human readable then XDR, it's still usually generated and consumed by computer programs.

I don't agree with this statement. In JS, JSON is actually a native representation of objects and is used quite often.

The biggest problem we need to solve is that a lot of boilerplate code is needed to create more complicated XDR objects. Check union example.

If it is meant for humans to interact with, maybe we can use TxREP (SEP11) as is?

FWIW as I've said in other tickets, I'm leaning towards not introducing more than one way to solve a problem, so if it's just a matter of giving developers a different way of assembling transactions, I'd opt to not do it.

It's not about transactions only. If you want to do any other task related to XDR (ex. decode Ledger contents encoded in XDR) it's actually really hard to do using existing js-xdr APIs.

[tomerweller]

In JS, JSON is actually a native representation of objects

I don't agree with that statement :) JSON doesn't capture the prototype functions (or class system, whatever JS people have decided on) which are heavily used in the current way that js-xdr generates these objects.

Looking at your union example I think I understand the problem better now and indeed it's not pretty. It seems like our generated xdr objects conflate the marshalling/unmarshalling code with the actual data ("OOP"). If there was a clear separation, it should be trivial to use simple JSON as input or output.

Does this make any sense? It seems like it will require a non-trivial amount of work to make that change though. @bartekn how important would you say is it?

[bartekn]

I don't agree with that statement :) JSON doesn't capture the prototype functions (or class system, whatever JS people have decided on) which are heavily used in the current way that js-xdr generates these objects.

I agree with what you've said 😉. What I was trying to say is that JSON is native to JS: you don't need to import a package or library to be able to marshal/unmarshal/traverse JSON, it's part of a language syntax. Because of this JS devs often use JSON syntax to create objects. Obviously if you want to add some behaviour to the object JSON is not enough but XDR is only used to transfer data.

I think my main problem with the current API is that you need to create objects bottom-up. TransactionBuilder in js-stellar-base is a good example. If you want to create a TransactionEnvelope you need to first create the leaf-objects of this structure (memo, operations, time bounds, etc.) and then add them to the higher level object. This is especially awkward when you want to create a union (see create_account operation code).

If it was possible to create XDR objects directly from JSON we could remove 50% of the code in js-stellar-base (the code responsible for building XDRs). Seriously.

It seems like it will require a non-trivial amount of work to make that change though. @bartekn how important would you say is it?

This is the issue I've been hearing very often from devs in our community. Some of them even created their own XDR packages to make this easier.

When it comes to difficulty of this task, XDR->JSON conversion is already done in XDR Viewer but instead of JSON we generate a HTML table. When it comes to JSON->XDR it will be probably some DFS-like algorithm. Maybe it's not trivial but it's also not an epic task, imho.

[tomquisel]

I think adding JSON -> XDR and XDR -> JSON converters to this library will greatly improve developer ergonomics.

@bartekn brought up a good example of how the current approach for creating XDR is awkward in JS in the union example. @vcarl also points out how awkward reading XDR in JS is in https://github.com/stellar/js-stellar-sdk/issues/209:

StellarSdk.xdr.TransactionResult.fromXDR(result.result_xdr, 'base64').result().value()[0].value().value().success().offer().switch().name

Instead, if the result were translated to JSON, I believe it could be as simple as: result[0].success.offer.name

You can see more examples of how rough it is to read from XDR in JS in https://github.com/stellar/js-stellar-sdk/pull/247.

[abuiles]

@bartekn @tomerweller @morleyzhi @tomquisel folks, could you give me your 5 cents on this? https://github.com/abuiles/json-xdr/issues/5 it tries to answers the question to How do you represent XDR union?