search

ponder-sh / ponder

A backend framework for crypto apps
https://ponder.sh
MIT License
613 stars 95 forks source link

Ponder should not shutdown due to the RPC failure of a specific chain. #861

Open jiguantong opened 5 months ago

jiguantong commented 5 months ago

It is inappropriate for the entire ponder service to go down when there is an issue with the RPC of a specific chain.

10:55:41 am ERROR process    Caught uncaughtException event with error:
Error: Start block number (2701388) cannot be greater than latest block number (71996).
         Are you sure the RPC endpoint is for the correct network?
    at validateHistoricalBlockRange (file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/sync-historical/validateHistoricalBlockRange.ts:13:11)
    at file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/sync-historical/service.ts:187:11
    at Array.map (<anonymous>)
    at HistoricalSyncService.setup (file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/sync-historical/service.ts:185:20)
    at file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/sync/service.ts:201:28
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Promise.all (index 1)
    at create4 (file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/sync/service.ts:166:55)
    at run (file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/bin/utils/run.ts:108:23)
    at start3 (file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/bin/commands/start.ts:64:23)
10:55:41 am WARN  process    Received uncaughtException, starting shutdown sequence
10:55:41 am FATAL process    Finished shutdown sequence, terminating (exit code 1)
error Command failed with exit code 1.
jiguantong commented 4 months ago

version: v0.4.2

1:29:53 pm INFO  historical Sync is 0.0% complete (contract=SignaturePub)
1:29:53 pm ERROR sync       Sync service failed during setup:
Error: Failed to fetch initial realtime data. (Hint: Most likely the result of an incapable RPC provider)
    at RealtimeSyncService.setup (file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/sync-realtime/service.ts:151:13)
    at file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/sync/service.ts:171:32
    at async Promise.all (index 0)
    at SyncService.start (file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/sync/service.ts:169:7)
    at run (file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/bin/utils/run.ts:252:31)
    at start (file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/bin/commands/start.ts:68:23)
    at Command.<anonymous> (file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/bin/ponder.ts:67:5)
    at Command.parseAsync (/root/msgport-git/ormponder/node_modules/commander/lib/command.js:1015:5)
    at file:///root/msgport-git/ormponder/node_modules/@ponder/core/src/bin/ponder.ts:123:1
1:29:53 pm WARN  process    Received fatal error, starting shutdown sequence
1:29:54 pm FATAL process    Finished shutdown sequence, terminating (exit code 1)

version: v0.4.35

2:49:52 pm INFO  historical Started syncing 'darwinia' logs for 'ORMPV2' with 0.0% cached
2:51:06 pm WARN  sync       Failed 'eth_getBlockByNumber' RPC request after 10 attempts
HttpRequestError: HTTP request failed.

URL: https://arbitrum-hrpc.vercel.app1/
Request body: {"method":"eth_getBlockByNumber","params":["latest",true]}

Details: fetch failed
Version: viem@1.21.3
    at Object.http (file:///root/msgport-git/ormponder/node_modules/viem/utils/rpc.ts:142:11)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at fn (file:///root/msgport-git/ormponder/node_modules/viem/clients/transports/http.ts:107:18)
    at request (file:///root/msgport-git/ormponder/node_modules/viem/clients/transports/http.ts:109:39)
    at withRetry.delay.count.count (file:///root/msgport-git/ormponder/node_modules/viem/utils/buildRequest.ts:129:18)
    at attemptRetry (file:///root/msgport-git/ormponder/node_modules/viem/utils/promise/withRetry.ts:37:22)
2:51:06 pm ERROR process    Caught uncaughtException event
HttpRequestError: HTTP request failed.

URL: https://arbitrum-hrpc.vercel.app1/
Request body: {"method":"eth_getBlockByNumber","params":["latest",true]}

Details: fetch failed
Version: viem@1.21.3
    at Object.http (file:///root/msgport-git/ormponder/node_modules/viem/utils/rpc.ts:142:11)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at fn (file:///root/msgport-git/ormponder/node_modules/viem/clients/transports/http.ts:107:18)
    at request (file:///root/msgport-git/ormponder/node_modules/viem/clients/transports/http.ts:109:39)
    at withRetry.delay.count.count (file:///root/msgport-git/ormponder/node_modules/viem/utils/buildRequest.ts:129:18)
    at attemptRetry (file:///root/msgport-git/ormponder/node_modules/viem/utils/promise/withRetry.ts:37:22)
2:51:06 pm WARN  process    Received uncaughtException, starting shutdown sequence
2:51:06 pm WARN  sync       Failed 'eth_chainId' RPC request after 10 attempts
HttpRequestError: HTTP request failed.

URL: https://arbitrum-hrpc.vercel.app1/
Request body: {"method":"eth_chainId"}

Details: fetch failed
Version: viem@1.21.3
    at Object.http (file:///root/msgport-git/ormponder/node_modules/viem/utils/rpc.ts:142:11)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at fn (file:///root/msgport-git/ormponder/node_modules/viem/clients/transports/http.ts:107:18)
    at request (file:///root/msgport-git/ormponder/node_modules/viem/clients/transports/http.ts:109:39)
    at withRetry.delay.count.count (file:///root/msgport-git/ormponder/node_modules/viem/utils/buildRequest.ts:129:18)
    at attemptRetry (file:///root/msgport-git/ormponder/node_modules/viem/utils/promise/withRetry.ts:37:22)
2:51:07 pm FATAL process    Finished shutdown sequence, terminating (exit code 1)
hackfisher commented 4 months ago

Related to https://github.com/ponder-sh/ponder/issues/869#issuecomment-2164957725

0xOlias commented 4 months ago

Hi @jiguantong, thanks for opening. The 0.4.35 logs you pasted above indicate that the https://arbitrum-hrpc.vercel.app1 RPC is not responding to the basic eth_chainId and eth_getBlockByNumber requests that Ponder sends on startup.

At the moment, it's not possible for Ponder to ignore such an error and continue indexing on the other chains, but that's a feature we are considering.

In the meantime, here are a few ways you can mitigate this:

  1. Switch to a more reliable RPC provider
  2. Use a Viem fallback transport and include additional RPC URLs that are more reliable
  3. If indexing that network is not critical to your application, remove it or comment it out of your ponder.config.ts temporarily.

Will leave this open to track progress on the feature request.