graphprotocol / graph-node

Graph Node indexes data from blockchains such as Ethereum and serves it over GraphQL
https://thegraph.com
Apache License 2.0
2.91k stars 971 forks source link

[Bug] No rpc metrics for non-`eth_getLogs` calls #4690

Open paymog opened 1 year ago

paymog commented 1 year ago

Bug report

I'm running graph node version 0.31.0 (recently upgraded from 0.29.0) and I don't see any metrics for rpc calls other than getLogs over the past month

image

I'm running 15+ indexers with ~3k subgraphs.

I tried checking my datadog agent to see if there's some rate limiting happening at the agent level and nothing stood out to me. I tried curl-ing the metrics endpoints for a few of my graph nodes, and none of them had non-eth_getLogs values. I see the following when I curl the metrics endpoint

# HELP eth_rpc_request_duration Measures eth rpc request duration
# TYPE eth_rpc_request_duration histogram
eth_rpc_request_duration_bucket{method="eth_getLogs",provider="arbitrum-one-default",le="0.05"} 597
eth_rpc_request_duration_bucket{method="eth_getLogs",provider="arbitrum-one-default",le="0.1"} 1224
eth_rpc_request_duration_bucket{method="eth_getLogs",provider="arbitrum-one-default",le="0.2"} 3323
eth_rpc_request_duration_bucket{method="eth_getLogs",provider="arbitrum-one-default",le="0.4"} 3506
eth_rpc_request_duration_bucket{method="eth_getLogs",provider="arbitrum-one-default",le="0.8"} 3631
eth_rpc_request_duration_bucket{method="eth_getLogs",provider="arbitrum-one-default",le="1.6"} 3634
eth_rpc_request_duration_bucket{method="eth_getLogs",provider="arbitrum-one-default",le="3.2"} 3634
eth_rpc_request_duration_bucket{method="eth_getLogs",provider="arbitrum-one-default",le="6.4"} 3634
eth_rpc_request_duration_bucket{method="eth_getLogs",provider="arbitrum-one-default",le="12.8"} 3634
eth_rpc_request_duration_bucket{method="eth_getLogs",provider="arbitrum-one-default",le="25.6"} 3634
eth_rpc_request_duration_bucket{method="eth_getLogs",provider="arbitrum-one-default",le="+Inf"} 3634
eth_rpc_request_duration_sum{method="eth_getLogs",provider="arbitrum-one-default"} 472.76402704999924
eth_rpc_request_duration_count{method="eth_getLogs",provider="arbitrum-one-default"} 3634

I did more digging and I found another metric, endpoint_request which isn't documented in the metrics doc which seems to have more up to date info like the following

# HELP endpoint_request successfull request
# TYPE endpoint_request counter
endpoint_request{conn_type="firehose",provider="goldsky-firehose-goerli",req_type="unknown",result="success"} 1859
endpoint_request{conn_type="firehose",provider="goldsky-firehose-mainnet",req_type="unknown",result="success"} 141
endpoint_request{conn_type="firehose",provider="streamingfast-firehose-bsc",req_type="unknown",result="success"} 23
endpoint_request{conn_type="firehose",provider="streamingfast-firehose-goerli",req_type="unknown",result="success"} 20
endpoint_request{conn_type="firehose",provider="streamingfast-firehose-mainnet",req_type="unknown",result="failure"} 1
endpoint_request{conn_type="firehose",provider="streamingfast-firehose-mainnet",req_type="unknown",result="success"} 228
endpoint_request{conn_type="firehose",provider="streamingfast-firehose-matic",req_type="unknown",result="success"} 25
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-archive",req_type="eth_getBlockByNumber",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-archive",req_type="net_version",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-archive",req_type="web3_clientVersion",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-default",req_type="eth_chainId",result="success"} 2
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-default",req_type="eth_getBlockByHash",result="failure"} 68
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-default",req_type="eth_getBlockByHash",result="success"} 9582
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-default",req_type="eth_getBlockByNumber",result="failure"} 26
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-default",req_type="eth_getBlockByNumber",result="success"} 3420
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-default",req_type="eth_getTransactionReceipt",result="failure"} 100
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-default",req_type="eth_getTransactionReceipt",result="success"} 24571
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-default",req_type="net_version",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-default",req_type="web3_clientVersion",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-traces",req_type="eth_getBlockByNumber",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-traces",req_type="net_version",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-goerli-traces",req_type="web3_clientVersion",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-nova",req_type="eth_getBlockByHash",result="success"} 8394
endpoint_request{conn_type="rpc",provider="arbitrum-nova",req_type="eth_getBlockByNumber",result="success"} 4065
endpoint_request{conn_type="rpc",provider="arbitrum-nova",req_type="eth_getTransactionReceipt",result="success"} 17247
endpoint_request{conn_type="rpc",provider="arbitrum-nova",req_type="net_version",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-nova",req_type="web3_clientVersion",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-nova-archive",req_type="eth_getBlockByNumber",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-nova-archive",req_type="net_version",result="success"} 1
endpoint_request{conn_type="rpc",provider="arbitrum-nova-archive",req_type="web3_clientVersion",result="success"} 1

Relevant log output

No response

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

OS information

None

github-actions[bot] commented 9 months ago

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.