I'm running a dYdX full node and currently using Cosmovisor to execute the binary. The problem is that I occasionally encounter errors like the one below, which causes the process to crash intermittently. Fortunately, it automatically restarts since it's running as a system service, but this still impacts the stability of the node's operation. I'd appreciate any help in resolving this issue!
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4679]: 7:06AM INF committed state block_app_hash=60E8FEDD65B60A7220DA8F8697AC85E1BA6D767C2AFEA28B111154109EDA6367 height=30686828 module=state
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4679]: 7:06AM INF indexed block events height=30686828 module=txindex
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4679]: 7:06AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=4 numToDial=6
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4679]: panic: slinky-client-price-fetcher-daemon unhealthy: no successful update has occurred; last failed update occurred at 2024-11-21 07:06:10.626649741 +0000 UTC m=+359.070293110 with error 'failed>
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4679]: goroutine 319709 [running]:
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4679]: github.com/dydxprotocol/v4-chain/protocol/daemons/server/types.(*HealthMonitor).RegisterService.PanicServiceNotResponding.func2({0x475dca0, 0xc00baf4ac0})
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4679]: github.com/dydxprotocol/v4-chain/protocol/daemons/server/types/health_monitor.go:191 +0x9e
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4679]: github.com/dydxprotocol/v4-chain/protocol/daemons/server/types.(*healthChecker).Poll(0xc0062fd130)
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4679]: github.com/dydxprotocol/v4-chain/protocol/daemons/server/types/health_checker.go:185 +0x91
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4679]: created by time.goFunc
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4679]: time/sleep.go:177 +0x2d
Nov 21 07:06:12 atn-dydx-mainnet-full1-de cosmovisor[4656]: Error: exit status 2
Nov 21 07:06:12 atn-dydx-mainnet-full1-de systemd[1]: cosmovisor.service: Main process exited, code=exited, status=1/FAILURE
Nov 21 07:06:12 atn-dydx-mainnet-full1-de systemd[1]: cosmovisor.service: Failed with result 'exit-code'.
Nov 21 07:06:12 atn-dydx-mainnet-full1-de systemd[1]: cosmovisor.service: Consumed 12min 57.656s CPU time.
Nov 21 07:06:13 atn-dydx-mainnet-full1-de systemd[1]: Stopped Cosmovisor daemon.
Nov 21 07:06:13 atn-dydx-mainnet-full1-de systemd[1]: cosmovisor.service: Consumed 12min 57.656s CPU time.
Nov 21 07:06:14 atn-dydx-mainnet-full1-de systemd[1]: Started Cosmovisor daemon.
linear[bot]
commented
1 week ago
I'm running a dYdX full node and currently using Cosmovisor to execute the binary. The problem is that I occasionally encounter errors like the one below, which causes the process to crash intermittently. Fortunately, it automatically restarts since it's running as a system service, but this still impacts the stability of the node's operation. I'd appreciate any help in resolving this issue!