Index-node core dumps when archive without traces is mentioned

trader-payne commented 3 years ago

Steps to reproduce

Have only one ETH archive node, without traces, and pass it to the index-node as following: ETHEREUM="mainnet:archive:"

Workaround

Pass the env vars as ETHEREUM="mainnet:", but then it automatically assumes it has traces even tho it's not mentioned, so it will spam the ETH RPC with useless requests

Logs

Oct 19 17:39:32.362 INFO Start subgraph, data_sources: 3, subgraph_id: QmcPSRpF28QSWpgAgNPFVyiGYd75L5aSrKGLXhmaFW5bSK, component: SubgraphInstanceManager
thread 'tokio-runtime-worker' panicked at 'expected eth adapter that matches subgraph network mainnet with required capabilities: archive, trace: ErrorMessage { msg: "A matching Ethereum network with NodeCapabilities { archive: true, traces: true } was not found." }', /graph-node/core/src/subgraph/instance_manager.rs:274:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Panic in tokio task, aborting!
/usr/local/bin/start: line 60:    26 Aborted                 (core dumped) graph-node --node-id "${node_id//-/_}" --postgres-url "$postgres_url" --ethereum-rpc $ethereum --ipfs "$ipfs"

Setup

I'm using a split graph-node, so index-node and query-node. Not sure if that matters, but figured it's a good idea to mention.

Notes

@alexo382 first encountered this

joesixpack commented 3 years ago

https://github.com/graphprotocol/graph-node/issues/1895

https://github.com/graphprotocol/graph-node/pull/1810

fattox commented 3 years ago

I've been doing this since the start while using Geth. It's been working fine for me. I set it this way when Ford first mentioned the feature, since i figured Geth will never do traces, so why not configure him that way from the start.

Can give more details if the team needs to narrow down the problem, since one of my nodes is still running this way even now: ETH_SRC="mainnet:archive:http://x.x.x.x:8545"

cargo run -p graph-node --release -- --ethereum-rpc $ETH_SRC \ ...

koen84 commented 3 years ago

I'm confirming this issue. Seems resolved removing the archive specification, yet then the agent might allocate subgraphs needing trace_* but the ETH node underneath won't be able to deliver that.

fordN commented 3 years ago

@trader-payne did you have an Ethereum node connected previously with traces enabled?

Right now a subgraph deployments will be gracefully rejected if no Ethereum node is found with the required capabilities, but I believe if a deployment is already assigned to a node, the subgraph or node restarts, and no Eth node is found with the required capabilities it causes this panic.

koen84 commented 3 years ago

I think this particular indexer of mine hasn't seen an ETH node with traces (i'm not sure), but it must have had tracing requiring subgraphs on it (that have gotten reassigned between indexers).

trader-payne commented 3 years ago

@fordN sorry for the (very) late response.

So the issues is as follows:

Have an index-node and assign an Ethereum archive node to it.

a) If you don't specify archive or traces in the arguments, eg. ETHEREUM="mainnet:http://x.x.x.x:port", everything works fine. Note - in the index-node, it shows that the capabilities for said archive node are both archive and traces without ever specifying that earlier, as mentioned. Kinda weird, but works just fine. If you assign subs with traces they'll spam RPC requests that fail, no biggie, but no core dump.

b) If you only specify archive in the arguments ETHEREUM="mainnet:archive:http://x.x.x.x:port" the index-node core dumps with the error mentioned in the OP. Referring back to a) note - I presume it panics because it's looking by default for both archive and traces and it doesn't find traces.

c) If you assign TWO different Ethereum nodes to THE SAME indexer, you can use whatever arguments you want and it will never panic.

Now, going back to what you mentioned, I'm pretty sure I had my index-node looking at both traces and non-traces subgraphs before encountering this issue, but honestly it shouldn't "remember" which capabilities your ETH node had in the past, and shouldn't core dump when only specifying archive.

I have only no-traces subgraphs on one of my index-nodes, I can test this more thoroughly tomorrow after I upgrade all the other components to the latest versions, and I'll report back once I have more conclusive results.

fordN commented 3 years ago

This issue should be resolved with https://github.com/graphprotocol/graph-node/pull/2014. I'll hold off on closing it until I have confirmation from indexers.

trader-payne commented 3 years ago

Love the name of that one, lol 😂

graphprotocol / mission-control-indexer