block polling failed in cronos mainnet

[PhantomTop]

I think this is the graph-node bug for not handling cronos blockchain network.

I built graph-node source and tried to connect cronos rpc server. The command is as follows.

cargo run -p graph-node --release -- --postgres-url postgresql://postgres:123456@localhost:5432/graph-node --ethereum-rpc cronosmainnet_25-1:https://evm-cronos.crypto.org --ipfs 127.0.0.1:5001 --debug

The result is as follows.

Finished release [optimized] target(s) in 0.16s
Running `target/release/graph-node --postgres-url 'postgresql://postgres:daniel@localhost:5432/graph-node' --ethereum-rpc 'cronosmainnet-25:https://evm-cronos.crypto.org' --ipfs '127.0.0.1:5001' --debug`
Dec 18 00:41:28.627 INFO Graph Node version: 0.25.0 :: fraction+22 (b1dfbafcf 2021-12-16)
Dec 18 00:41:28.627 INFO Generating configuration from command line arguments
Dec 18 00:41:28.627 INFO Starting up
Dec 18 00:41:28.627 INFO Trying IPFS node at: http://127.0.0.1:5001/
Dec 18 00:41:28.632 INFO Creating transport, capabilities: archive, traces, url: https://evm-cronos.crypto.org, provider: cronosmainnet-25-rpc-0
Dec 18 00:41:28.632 INFO Successfully connected to IPFS node at: http://127.0.0.1:5001/
Dec 18 00:41:28.880 DEBG Creating firehose networks [1 chains, ingestor default]
Dec 18 00:41:28.880 DEBG Cleaning up large notifications after about 300s, channel: store_events, component: NotificationListener
Dec 18 00:41:28.888 INFO Connecting to Postgres, weight: 1, conn_pool_size: 10, url: postgresql://postgres:HIDDEN_PASSWORD@localhost:5432/graph-node, pool: main, shard: primary
Dec 18 00:41:28.890 INFO Pool successfully connected to Postgres, pool: main, shard: primary, component: Store
Dec 18 00:41:28.890 DEBG Using postgres host order [Main], shard: primary, component: Store
Dec 18 00:41:28.899 INFO Setting up fdw, pool: main, shard: primary, component: ConnectionPool
Dec 18 00:41:28.901 INFO Running migrations, pool: main, shard: primary, component: ConnectionPool
Dec 18 00:41:28.904 INFO Migrations finished, pool: main, shard: primary, component: ConnectionPool
Dec 18 00:41:28.904 INFO Mapping primary, pool: main, shard: primary, component: ConnectionPool
Dec 18 00:41:28.930 DEBG Cleaning up large notifications after about 300s, channel: chain_head_updates, component: ChainHeadUpdateListener > NotificationListener
Dec 18 00:41:28.937 INFO Connecting to Ethereum to get network identifier, capabilities: archive, traces, provider: cronosmainnet-25-rpc-0
Dec 18 00:41:29.321 INFO Connected to Ethereum, capabilities: archive, traces, network_version: 25, provider: cronosmainnet-25-rpc-0
Dec 18 00:41:29.375 INFO Creating LoadManager in disabled mode, component: LoadManager
Dec 18 00:41:29.376 INFO Starting block ingestors with 1 chains [cronosmainnet-25]
Dec 18 00:41:29.376 INFO Starting block ingestor for network, network_name: cronosmainnet-25
Dec 18 00:41:29.376 INFO Starting firehose block ingestors with 0 chains []
Dec 18 00:41:29.376 INFO Starting job runner with 4 jobs, component: JobRunner
Dec 18 00:41:29.376 DEBG Schedule for Vacuum subgraphs.subgraph_deployment, first_run_in_s: 0, interval_s: 60, component: JobRunner
Dec 18 00:41:29.376 DEBG Schedule for Report pg_notification_queue_usage(), first_run_in_s: 0, interval_s: 60, component: JobRunner
Dec 18 00:41:29.376 DEBG Schedule for Reconcile important tables from the primary, first_run_in_s: 9, interval_s: 900, component: JobRunner
Dec 18 00:41:29.377 DEBG Schedule for Record and remove unused deployments, first_run_in_s: 79, interval_s: 7200, component: JobRunner
Dec 18 00:41:29.378 INFO Starting JSON-RPC admin server at: http://localhost:8020, component: JsonRpcServer
Dec 18 00:41:29.378 INFO Started all assigned subgraphs, node_id: default, count: 0, component: SubgraphRegistrar
Dec 18 00:41:29.379 INFO Starting GraphQL HTTP server at: http://localhost:8000, component: GraphQLServer
Dec 18 00:41:29.379 INFO Starting index node server at: http://localhost:8030, component: IndexNodeServer
Dec 18 00:41:29.379 INFO Starting metrics server at: http://localhost:8040, component: MetricsServer
Dec 18 00:41:29.380 INFO Starting GraphQL WebSocket server at: ws://localhost:8001, component: SubscriptionServer
Dec 18 00:41:29.573 INFO Downloading latest blocks from Ethereum. This may take a few minutes..., provider: cronosmainnet-25-rpc-0, component: BlockIngestor
Dec 18 00:41:30.412 WARN Trying again after block polling failed: 0x87fe…51f2, provider: cronosmainnet-25-rpc-0, component: BlockIngestor
Dec 18 00:41:31.609 INFO Downloading latest blocks from Ethereum. This may take a few minutes..., provider: cronosmainnet-25-rpc-0, component: BlockIngestor
Dec 18 00:41:32.808 WARN Trying again after block polling failed: 0x87fe…51f2, provider: cronosmainnet-25-rpc-0, component: BlockIngestor
Dec 18 00:41:34.032 INFO Downloading latest blocks from Ethereum. This may take a few minutes..., provider: cronosmainnet-25-rpc-0, component: BlockIngestor
Dec 18 00:41:35.906 WARN Trying again after block polling failed: 0xa031…de10, provider: cronosmainnet-25-rpc-0, component: BlockIngestor

Help me, please.

[maoueh]

I would assume something is not working as expected in the RPC, that would be my first guess. It's unclear which code path inside graph-node lead to the Trying again after block polling failed: 0xa031…de10 message.

Could you run with GRAPH_LOG=trace to see more details.

[PhantomTop]

The error string is as follows. Dec 26 21:27:34.562 WARN Trying again after eth_getBlockByNumber(0, false) RPC call failed (attempt #10) with result Err(RPC error: Error { code: ServerError(-32000), message: "rpc error: code = Unknown desc = panic message redacted to hide potentially sensitive system info: panic", data: None }

Caused by: Server error: rpc error: code = Unknown desc = panic message redacted to hide potentially sensitive system info: panic), provider: mainnet-rpc-0 Dec 26 21:27:35.305 WARN Trying again after eth_getBlockByNumber(0, false) RPC call failed (attempt #11) with result Err(RPC error: Error { code: ServerError(-32000), message: "rpc error: code = Unknown desc = panic message redacted to hide potentially sensitive system info: panic", data: None }

Caused by: Server error: rpc error: code = Unknown desc = panic message redacted to hide potentially sensitive system info: panic), provider: mainnet-rpc-0 Dec 26 21:27:36.994 WARN Trying again after eth_getBlockByNumber(0, false) RPC call failed (attempt #12) with result Err(RPC error: Error { code: ServerError(-32000), message: "rpc error: code = Unknown desc = panic message redacted to hide potentially sensitive system info: panic", data: None }

When I start cronos local node from the beginning, graph-node connecting to the local node works well. But after a few hours it doesn't connect to the cronos node. What's the reason?

[azf20]

@PhantomTop that looks like there is an issue with the Cronos RPC, responding to this RPC call:

curl --location --request POST 'https://evm-cronos.crypto.org' \
--header 'Content-Type: application/json' \
--data-raw '{
    "jsonrpc":"2.0",
    "method":"eth_getBlockByNumber",
    "params":[
        "0", 
        false
    ],
    "id":1
}'

If this issue is intermittent (as you say it only occurs after a few hours), then you may want to check with the RPC provider. Closing this issue as it appears to be an RPC issue. Thanks!