Architecture discussion: JSON RPC API specification

[steveluscher]

Problem

We've long struggled with having the implementation of the JSON RPC server fall out of sync with the clients (eg. the @solana/web3.js Typescript client). This happens because the implementations are hand-rolled.

The effort to fully specify the network gives us the perfect opportunity to eliminate all sources of manual error/omission.

Proposed solution

1. Hand-write a machine-readable specification artifact, here, that fully describes the complete JSON RPC interface in terms of:
   • the names of its methods
   • their input types
   • their output types
   • their deprecation status
   • their membership in the ‘minimal’ or ‘full’ RPC method set
2. Code generate artifacts from that specification artifact:
   • the Rust interface (like the present-day rpc/src/rpc.rs)
   • the Typescript interface (like the present-day method signatures in web3.js/src/connection.ts)
   • assertions for use in automated tests (eg. assertions about the expected JSON output from a given method)

The specification artifact would live in this repository. Implementors would import that specification and use it as part of a codegen scheme that produces the kinds of artifacts described above, in a format that makes sense for their codebase.

Inspiration

We can take inspiration from schema definition languages like OpenRPC, but it's my opinion that we will have to ultimately develop our own. In particular, our extant RPC features runtime behaviours that can change the output types depending on the input to an RPC method.

Take for example, the getBlock method. If you pass transactionDetails: 'full' then the output takes this form:

type GetBlockMethodResponse = {
  transactions: Array<{
    transaction: {
      message: VersionedMessage;
      signatures: string[];
    };
  }>;
}

If, however, you pass transactionDetails: 'accounts' then the output takes this form:

type GetBlockMethodResponse = {
  transactions: Array<{
    transaction: {
      accountsKeys: ParsedMessageAccount[],
      signatures: string[];
    };
  }>;
}

For this reason, we need to be able to specify multiple override signatures per JSON RPC method – something that I don't believe OpenRPC supports (see where their docs state: “The method name is used as the method field of the JSON-RPC body. It therefore MUST be unique” – issue).

[2501babe]

cursory googling is failing me because typescript has gone through so many type system changes that a lot of information seems outdated, but...

if typescript supports sum types (rust enums, c tagged unions) somehow, we could encode the different output formats at the type level that way

edit: actually i guess that would break backwards compat for consumers of the client library, but... honestly the mechanism you described to me in dms where it sounds like it infers the concrete type at runtime based on input arguments? is probably worth breaking. perhaps do it in a separate namespace and deprecate the old one eventually?

edit2: we would also need to redesign the rust types i think because as best i can tell the different output formats are encoded very deep in the struct on an ad hoc basis. but im still coming up to speed on how all the rpc machinery works

[BelfordZ]

Hey, I wrote the open rpc spec and I work on a lot the tooling. I'm happy to chat, as I'd love for openrpc to be able to accomodate your needs.

I think there is a feature we can borrow from open api that would ease your issue - discriminators. Please see here for more.

I initially omitted this feature when we forked off of openapi because the interface was a bit cumbersome, and at the time, there was no one screaming from the hilltops asking for discriminators to work.

Anyways, feel free to open an issue with openrpc spec to help you address your concerns. If not, and you chose to go your own path, I will be eagerly watching to see what great stuff you guys come up with, and hopefully incorperate those into OpenRPC.

[etodanik]

@BelfordZ I'm actually making a codegen project for Solana that uses discriminators with a fork of OpenAPI just as an experiment. It'd be amazing to have it as part of the spec.

[BelfordZ]

@etodanik very cool!! I'd love to take a look if possible. Any chance one of you guys would like to put together a proposal PR to https://github.com/open-rpc/spec ? Im happy to jump in a call and talk it over with anyone interested.