Look into gRPC / Tendermint RPC for Cosmos SDK 0.45+

ethanfrey commented 3 years ago

They seem to make some optimisations such that the gRPC server will not longer just proxy through Tendermint. To increase concurrency I believe. This may mean it would be more sensible to call the gRPC server than Tendermint RPC. Maybe...

Anyway, something to look into when we add SDK 0.45+ support

https://github.com/cosmos/cosmos-sdk/pull/10045#issuecomment-930156775

tac0turtle commented 2 years ago

this would be awesome. if grpc support is added we could neat things with grpc reflection to get possible queries and txs of a chain.

webmaster128 commented 2 years ago

We use the following functions from the Tendermint34Client / Tendermint RPC:

[x] status for chain ID: https://github.com/cosmos/cosmos-sdk/blob/release/v0.45.x/proto/cosmos/base/tendermint/v1beta1/query.proto#L17-L19
[x] status for height: https://github.com/cosmos/cosmos-sdk/blob/release/v0.45.x/proto/cosmos/base/tendermint/v1beta1/query.proto#L24-L27 (maybe a bit overhead but we get what we need)
[x] abciQuery without proofs
[x] abciQuery with proofs (Maybe: https://github.com/cosmos/cosmos-sdk/issues/7036?)
[ ] blockchain (block headers from minHeight to maxHeight)
[x] block (full block at height): https://github.com/cosmos/cosmos-sdk/blob/release/v0.45.x/proto/cosmos/base/tendermint/v1beta1/query.proto#L28-L31
[x] broadcastTxSync: https://github.com/cosmos/cosmos-sdk/blob/release/v0.45.x/proto/cosmos/tx/v1beta1/service.proto#L26-L32
[ ] txSearch with arbitrary query
[ ] Subscriptions: Block headers (to get next block ASAP for proof verification)

The ones checked I found in the gRPC API. Any idea about the others?

webmaster128 commented 2 years ago

@marbar3778 can you help with any item from the list above to make progress here?

tac0turtle commented 2 years ago

Sorry I looked into some of this but never responded. I think grpc should not be a replacement for rpc right now. But instead a way to at runtime interact with a chain without preexisting knowledge.

With grpc there is no need for abciQuery without proof. Tx search and subscriptions should still be handled by the rpc, but txsearch has a path to deprecation so it should not be relied on instead the app developer should use a Postgres instance with the psql indexer on tendermint.

webmaster128 commented 2 years ago

Thanks for the insights.

But instead a way to at runtime interact with a chain without preexisting knowledge.

All we do in CosmJS is statically typed. I wonder how this is envisioned to work. Even if we dynamically get available queries and custom message types from the chain, you need a code generator generating all the proto code for those types.

tac0turtle commented 2 years ago

this is the best example code I have seen so far. https://github.com/fdymylja/dynamic-cosmos. maybe @fdymylja could touch on this?

fdymylja commented 2 years ago

Thanks for the insights.

But instead a way to at runtime interact with a chain without preexisting knowledge.

All we do in CosmJS is statically typed. I wonder how this is envisioned to work. Even if we dynamically get available queries and custom message types from the chain, you need a code generator generating all the proto code for those types.

Not familiar with JS protobuf, but there should be a protoreflection implementation available.

And even if there isn't it shouldn't be too difficult to create. Once you get information on how a protobuf type looks like (what we care about is field tags + field kinds in order to know how to wire-encode them), you can work with any type as if it was a map\<string, ArbitraryType>.

Then once https://github.com/cosmos/cosmos-sdk/pull/10977 is merged, you won't need to keep track of message signers anymore.

If this is something you are planning to support let me know and I'll try to help.

webmaster128 commented 2 years ago

Thank you for the pointers. protobuf.js (unmaintained) supports the dynamic type generation at runtime.

I just wonder why you would want all of this in the first place? The CosmJS users we see know exaxtly what their backend is and what is available thete. For them the abstraction of CosmJS is already overwhelming. Why move all the compile time safety we have right now and make it runtime problems?

fdymylja commented 2 years ago

Thank you for the pointers. protobuf.js (unmaintained) supports the dynamic type generation at runtime.

I just wonder why you would want all of this in the first place? The CosmJS users we see know exaxtly what their backend is and what is available thete. For them the abstraction of CosmJS is already overwhelming. Why move all the compile time safety we have right now and make it runtime problems?

I'm not super familiar with cosmJS and especially how it is used by its consumers.

Why move all the compile time safety we have right now and make it runtime problems?

You can work both in a typed and untyped way.

In golang clients we have certain use-cases that are unlocked through reflection (also because sdk's golang client is not really a client but something that is totally chain-specific and revolves around the CLI).

Allow to work with a chain's custom modules (typed or untyped)
Allow to work with multiple sdk chains at the same time (typed or untyped)
Dynamic derivation of account address
Dynamic fetching of signer infos
Dynamic resolution of google.Protobuf.Any, ex: chain specific custom governance proposal, authz module authorisations.
It improves the flow of signing tremendously.
Allow to verify/sign things.
everything can be jsonified without previous knowledge on types

For example possible use cases in a frontend:

talk with any Cosmos SDK chain and send any msg to it
human readable store exploration
human readable query exploration (supports chain's custom types such as gov proposals etc)

You could build a sort of etherscan in cosmos in which instead of dynamically querying contracts you can query module queriers.

Same goes for tx messages.

IDK if this is something that cosmjs supports/or wants to support, but as I said if you see value in it, let me know and I'll be happy to support.

webmaster128 commented 2 years ago

Thanks a lot for taking the time to describe all of this, @fdymylja! I'll have a think about it to see if this is useful for our context and maybe create a draft implementation independent of CosmJS.

One thing that is clear is that using reflection and connecting to the gRPC endpoint should be discussed separately.

Cashmaney commented 2 years ago

Adding my 2 cents here.

The switch to GRPC/RPC makes it much harder for infrastructure providers to perform smart caching and advanced load-balancing/rate limiting on requests, due to the more complex nature of the requests.

For example, on current Secret Network mainnet when applying caching for traffic generated by Keplr we save about 50% of requests, since a lot of them go to endpoints that don't change much (voting, chain parameters, etc) - this has a compounding effect on response times, since it both returns requests much faster, and frees up resources for more intensive requests.

Personally I think it makes the most sense for cosmjs as a frontend library to support the GRPC-gateway endpoints. Not only are they easier for developers and infra providers to work with, but having requests not connection-orriented is an advantage, since it allows for better load-balancing between multiple servers. Lastly, working with the GRPC-gateway endpoints is much simpler since the swagger file is generated automagically from the proto defs.

I'm not enitrely familiar with how everything is hooked up behind the scenes on in the sdk, though, so I can only speak from the consumer side of things

nooomski commented 2 years ago

Love this conversation ❤️

To add to, or perhaps support what @fdymylja is saying, in terms of users: the multi-chain application developer is definitely one of them, especially as we move forward towards a 1000+ chain ecosystem.

More specifically, that would be wallets who'd want to extend functionality, or applications that use core features available to most up-to-date chains, most notably Authz, Gov, Groups, Feegrant. All all of these allow you to take a Msg as an input to perform an action. Think DAOs being able to trade on Osmosis without Osmosis having to build a custom app, validators authorizing an account to perform a chain's specific modified staking Tx without that chain building a custom app for that, one multi-chain gov tool to create proposals, etc, all without using the CLI. This is currently not possible and I see a strong need for it.

Just commenting here because I'm personally quite interested in getting this to work in the long term, and would have all 4 front-end projects that I'm currently involved in likely benefit from this.

I can't comment much in terms of what its place would be inside CosmJS. I recognise that mixing this in with current CosmJS would confuse users in terms of where they can expect errors (runtime v.s. compilation). If anything, I can see @cosmjs/reflection being a separate CosmJS package that would need some good documentation about usage and risks, and I would be happy to help on the docs side.

tac0turtle commented 1 year ago

adding the option to use grpc but not a replacement is easy enough and shouldnt have any requirements, surprised it hasnt happened yet.

ethanfrey commented 1 year ago

@tac0turtle please see https://github.com/cosmos/cosmjs/issues/885#issuecomment-1009370262

It seems there were two calls that we didn't have access to. It would be great for you and Simon to discuss this to find a proper resolution

ethanfrey commented 1 year ago

I'm also surprised Confio has not received any funding for this yet...

It is listed as a Q3 goal in our 2023 ICF grant proposal. If we get our Q3 plan approved, it should be done then. It would be great if there is a plan for the best possible CosmJS and Cosmos SDK integration before then

cosmos / cosmjs

Look into gRPC / Tendermint RPC for Cosmos SDK 0.45+ #885