graphprotocol / graph-tooling

Monorepo for various tools used by subgraph developers.
https://thegraph.com/docs
Apache License 2.0
386 stars 206 forks source link

How to recover one failed subgraph? #1102

Closed lansehuiyi6 closed 1 year ago

lansehuiyi6 commented 2 years ago

Hi, I want to know how to recover one failed healthy subgraph if subgraph sync failed because of rpc issue? For now, i just re-deploy one, and wait it sync to latest block. Any better idea?

Connection checkout timed out, backtrace: 0: ::handle_timeout 1: r2d2::Pool::get_timeout 2: graph_store_postgres::connection_pool::PoolInner::get_with_timeout_warning 3: graph_store_postgres::deployment_store::DeploymentStore::get_replica_conn 4: ::find_query_values 5: graph_graphql::store::prefetch::execute_field 6: graph_graphql::store::prefetch::execute_selection_set 7: graph_graphql::store::prefetch::run 8: ::prefetch 9: graph_graphql::execution::execution::execute_root_selection_set_uncached 10: <tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll 11: tokio::runtime::task::harness::Harness<T,S>::poll 12: tokio::runtime::blocking::pool::Inner::run 13: std::sys_common::backtrace::__rust_begin_short_backtrace 14: core::ops::function::FnOnce::call_once{{vtable.shim}} 15: std::sys::unix::thread::Thread::new::thread_start 16: start_thread 17: clone , wait_ms: 5000, pool: main, shard: primary, component: ConnectionPool

WARN Trying again after block polling failed: Transport error: Unexpected response status code: 429 Too Many Requests, provider: mainnet-rpc-0, component: BlockIngestor

schmidsi commented 2 years ago

I assume you are running the graph-node by yourself, right? The latest graph-node automatically retries the in-deterministically failed subgraphs. Are you running the latest graph-node?

lansehuiyi6 commented 2 years ago

I assume you are running the graph-node by yourself, right? The latest graph-node automatically retries the in-deterministically failed subgraphs. Are you running the latest graph-node?

Yes, i run node myself, not the latest one. I'll try this. Btw, which version had this automatically retry been added?

leoyvens commented 2 years ago

@lansehuiyi6 this is on master but not yet on a released version.

azf20 commented 1 year ago

Automatic retries are available for non-deterministic errors