[experimental] Discussion: Extending an RPC with custom methods

steveluscher commented 9 months ago

Motivation

We've written a comprehensive typespec for the Solana JSON-RPC in rpc-core. If you're using an RPC that offers more than the methods we've specified, you might like to extend your rpc object to include those extra methods.

Example use case

Maybe you use QuickNode and you want to make use of the qn_fetchNFTCollectionDetails_v2 method. You can easily mix a typespec for qn_fetchNFTCollectionDetails_v2() into the existing Solana ones, just like we do here:

https://github.com/solana-labs/solana-web3.js/blob/fc4e94336e635b203374dd40cc62ed8248774115/packages/rpc-core/src/rpc-subscriptions/index.ts#L80-L86

Problem

While our current implementation lets you:

augment the TypeScript types with custom methods
add pre/post processors to the inputs/results by supplying a concrete api implementation for your custom method (eg. you could call the QuickNode method above fetchNFTCollectionDetails() and have a concrete implementation substitute the JSON-RPC method name qn_fetchNFTCollectionDetails_v2).

…what we can't do with the current implementation is to customize anything about the transport. This is a problem for, for instance, Helius. Helius doesn't implement custom APIs via proprietary JSON-RPC methods, but rather offers completely different endpoints for those APIs (eg. https://api.helius.xyz/v0/addresses/<address>/balances?api-key=<your-key>).

Question

Does this matter? Is it a code smell to have IRpcApi know anything about transports? If an API is offered at a different endpoint should it by definition require that you create a new rpc object with a new RpcTransport?

buffalojoec commented 9 months ago

Our spec still requires your alternate JSON RPC endpoint to adhere to this payload structure though, right?

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "getAccountInfo",
  "params": [
    "vines1vzrYbzLMRdu58ou5XTby4qAqVRLmqo36NKPTg",
    {
      "encoding": "base58"
    }
  ]
}

We're crafting this underneath createJsonRpc(..):

https://github.com/solana-labs/solana-web3.js/blob/c7ef49cc49ee61422a4777d439a814160f6d7ce4/packages/rpc-transport/src/json-rpc.ts#L18

https://github.com/solana-labs/solana-web3.js/blob/c7ef49cc49ee61422a4777d439a814160f6d7ce4/packages/rpc-transport/src/json-rpc-message.ts#L1-L10

If someone like QuickNode had a different payload format, they'd need to roll their own JSON RPC so they could provide createQuickNodeJsonRpc(..) as an argument to createQuickNodeRpc(..), where createQuickNodeRpc(..) is analogous to createSolanaRpc(..), and maybe they include our methods alongside theirs or don't.

In the case of Helius (using URL routes and parameters), they do something similar to craft their payload alongside any of our methods (or QuickNode's) that they want to support in the createHeliusRpc(..) return type.

Either way, this amount of work is starting to look more like "building an RPC library using the types from Web3JS" rather than "injecting my plugin into Web3JS".

Perhaps we need to spec this out and make things more flexible?

buffalojoec commented 9 months ago

Does this matter? Is it a code smell to have IRpcApi know anything about transports?

FWIW I think with the way it's intended to be designed, it might be a bit of a code smell. Ideally you want to provide Solana-JSON-RPC-specific stuff (like the JSON RPC message in my above comment) from the main high-level library.

In other words, whatever I inject into @solana/rpc-transport should be the total basket of goods I require to use @solana/rpc-transport on my Solana JSON RPC.

Similarly, I should be able to write only a high-level library and (maybe) my own @helius/rpc-core libraries and then inject the necessary things into @solana/rpc-transport to use all of the things that it comes with.

If an API is offered at a different endpoint should it by definition require that you create a new rpc object with a new RpcTransport?

If I write my own API and then I have to write my own transport, what's the main benefit here?

steveluscher commented 9 months ago

If someone like QuickNode had a different payload format, they'd need to roll their own JSON RPC…

Oh, yeah this would only work for people who implement actual JSON-RPC 2.0, which is well specified. No different payload formats allowed.

lorisleiva commented 8 months ago

Please correct me if I'm wrong but here's my understanding of RPC transports and APIs semantically:

RPC Transports: "Give me some payload and I'll give you the requested response. Don't worry about how this will happen, I'll take care of that for you".
RPC API: "Here's a set of payload/response pairs we support".

Ideally, the former is customisable by the end-user of this library whereas the latter would typically be provided by the RPC provider.

If an RPC provider needs full control over the RPC Transport for it to work, then I'd say it's their responsibility to provide both the transport and the API which unfortunately, doesn't leave much room for the end-user to customise their RPC Transport.

However, the Helius example provided in the original message doesn't fit in this category for me. If the endpoint is likely to be modified by the API (here, appending some URL segments and query parameters), then we may need a urlTransformer the same way @buffalojoec is adding a paramsTransformer and a responseTransformer to the RPC in PR #1781.

Does that make sense or am I completely missing the point? 😅

EDIT: Since the RPC API is already communicating to the RPC Transport using a responseProcessor, we could add a requestProcessor to do the same on the opposite direction. (nit: I think requestInterceptor and responseInterceptor are better suited names here).

buffalojoec commented 8 months ago

RPC Transports: "Give me some payload and I'll give you the requested response. Don't worry about how this will happen, I'll take care of that for you".

RPC API: "Here's a set of payload/response pairs we support".

Ideally, the former is customisable by the end-user of this library whereas the latter would typically be provided by the RPC provider.

Yep this is how I view the relationship, too. Did you roll your own methods? Simply concatenate them to the Solana RPC (or don't, I don't care) and release your full type spec for users to plug into Web3 JS.

If an RPC provider needs full control over the RPC Transport for it to work, then I'd say it's their responsibility to provide both the transport and the API which unfortunately, doesn't leave much room for the end-user to customise their RPC Transport.

I think this is exactly the trade-off you make when you decide to go with URL params instead of a request body. However, what we have to hash out is: does using URL parameters mean you're not implementing the JSON RPC spec? I'm not convinced it means you're non-compliant.

If it means you're still compliant with the spec, then we should probably have a pluggable way for you to customize the transport for your URL structure(s). So, what you've proposed is probably the best way to do that IMO.

then we may need a urlTransformer

👆🏼

(nit: I think requestInterceptor and responseInterceptor are better suited names here)

I don't hate these but don't they sound a little malicious? 😅

solana-labs / solana-web3.js