Open exp0nge opened 9 months ago
Possibly related to #2415, #2726.
Have you tried running your node in debug mode (temporarily) and checking which RPC endpoints are being queried?
You can do so by setting this in your config.toml
:
# Output level for logging, including package level options
log_level = "debug"
Please note that you may need to set up additional software in case you are exposing your node to the public, such as a load balancer or a proxy, much like you would if you were exposing any other backend service.
Possibly related to #2415, #2726.
Have you tried running your node in debug mode (temporarily) and checking which RPC endpoints are being queried? You can do so by setting this in your
config.toml
:# Output level for logging, including package level options log_level = "debug"
Please note that you may need to set up additional software in case you are exposing your node to the public, such as a load balancer or a proxy, much like you would if you were exposing any other backend service.
The following gRPC requests are used:
In particular GetBlockByHeightRequest
is used heavily alongside GetTxRequest
(I am switching over to GetEvents to batch the per tx call)
Possibly related to #2415, #2726.
Have you tried running your node in debug mode (temporarily) and checking which RPC endpoints are being queried? You can do so by setting this in your
config.toml
:# Output level for logging, including package level options log_level = "debug"
Please note that you may need to set up additional software in case you are exposing your node to the public, such as a load balancer or a proxy, much like you would if you were exposing any other backend service.
By the way, I read through some of the suggestions in https://github.com/cosmos/gaia/issues/2415, but this is a perf. regression. We have solved in creative ways, but the nodes consistently die.
edit: We trust Interchain's/this repo's interpretation of the blockchain state, adding a layer of indrection like an explorer, while possible, is something we would very much like to avoid.
edit: We trust Interchain's/this repo's interpretation of the blockchain state, adding a layer of indrection like an explorer, while possible, is something we would very much like to avoid.
Indeed @exp0nge , every layer and redirection adds overhead and complexity - thats something that we always want to avoid if possible. The performance bottlenecks that you've mentioned, regression or otherwise, lie with the lower levels of the stack (SDK / Comet). If theres a specific endpoint that has regressed in terms of performance, then we can check with the relevant teams. Are the following endpoints GetBlockByHeightRequest
& GetTxRequest
causing most problems?
edit: We trust Interchain's/this repo's interpretation of the blockchain state, adding a layer of indrection like an explorer, while possible, is something we would very much like to avoid.
Indeed @exp0nge , every layer and redirection adds overhead and complexity - thats something that we always want to avoid if possible. The performance bottlenecks that you've mentioned, regression or otherwise, lie with the lower levels of the stack (SDK / Comet). If theres a specific endpoint that has regressed in terms of performance, then we can check with the relevant teams. Are the following endpoints
GetBlockByHeightRequest
&GetTxRequest
causing most problems?
Unsure exactly which call causes this as we use the ones above listed, but the most heavily used ones are: tx.v1beta1: BroadcastTxRequest, GetTxRequest
@exp0nge Hey, has this issue been fixed on v15.1.0
for your usecase?
Summary of Bug
We have noticed severe performance degradation since v11 where nodes we query against fall behind very quickly. Attached below are some logs. This behavior is not notice-able in non-RPC mode nodes (nodes that are not being queried).
cc @ckarabats
Version
>=v11
Steps to Reproduce
For Admin Use