Network subgraph gets removed

aasseman commented 1 year ago

Versions tested: v0.20.20, v0.20.21

At indexer-agent start, the network subgraph gets added to graph-node, then removed shortly after.

I selected only the relevant logs:

{"level":30,"time":1692911987139,"pid":1,"hostname":"indexer-agent-0","name":"IndexerAgent","component":"Network","indexer":"0xd75C4Dbcb215A6cf9097cfbCc70aAB2596B96a9C","protocolNetwork":"eip155:1","component":"NetworkSubgraph","endpoint":"https://gateway.thegraph.com/network","deployment":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs","msg":"Network subgraph is synced 63.76% (block #11468965 of #17987197)"}
{"level":30,"time":1692911992555,"pid":1,"hostname":"indexer-agent-0","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/5EFZk62JWs","msg":"Create subgraph name"}
{"level":30,"time":1692911992740,"pid":1,"hostname":"indexer-agent-0","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/5EFZk62JWs","msg":"Successfully created subgraph name"}
{"level":30,"time":1692911992741,"pid":1,"hostname":"indexer-agent-0","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/5EFZk62JWs","deployment":{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"},"node_id":"index-node-2","msg":"Deploy subgraph deployment"}
{"level":30,"time":1692911992741,"pid":1,"hostname":"indexer-agent-0","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/5EFZk62JWs","deployment":{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"},"node_id":"index-node-2","msg":"Deploy subgraph deployment"}
{"level":30,"time":1692911993327,"pid":1,"hostname":"indexer-agent-0","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/5EFZk62JWs","deployment":{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"},"msg":"Successfully deployed subgraph deployment"}
{"level":30,"time":1692911993327,"pid":1,"hostname":"indexer-agent-0","name":"IndexerAgent","component":"GraphNode","deployment":{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"},"node":"index-node-2","msg":"Reassign subgraph deployment"}
{"level":30,"time":1692912046968,"pid":1,"hostname":"indexer-agent-0","name":"IndexerAgent","component":"Network","indexer":"0xd75C4Dbcb215A6cf9097cfbCc70aAB2596B96a9C","protocolNetwork":"eip155:1","component":"NetworkSubgraph","endpoint":"https://gateway.thegraph.com/network","deployment":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs","msg":"Network subgraph is synced 63.76% (block #11468965 of #17987202)"}
{"level":30,"time":1692912051361,"pid":1,"hostname":"indexer-agent-0","name":"IndexerAgent","component":"Agent","deploy":[],"remove":[{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"}],"msg":"Deployment changes"}
{"level":30,"time":1692912051439,"pid":1,"hostname":"indexer-agent-0","name":"IndexerAgent","component":"GraphNode","deployment":{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"},"msg":"Remove subgraph deployment"}
{"level":30,"time":1692912051449,"pid":1,"hostname":"indexer-agent-0","name":"IndexerAgent","component":"GraphNode","deployment":{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"},"msg":"Successfully removed subgraph deployment"}

cryptovestor21 commented 1 year ago

+1, observed on mainnet with 20.20 in multimode

sbond14 commented 1 year ago

Observing the same issue on v0.20.21 single network mode

tilacog commented 1 year ago

Here are two possible solutions:

The Agent can automatically generate an indexing rule for the network subgraph during startup, setting decisionBasis to always.
- If there's a configuration change replacing the network subgraph deployment, this rule will also be updated.
- We can indicate this rule's association with the network subgraph in the custom column of the IndexingRules table.
Simply remove the network subgraph deployment ID from the step in the reconciliation loop that identifies deployments for removal.

tilacog commented 1 year ago

Please disregard my previous message.

I see that the Agent already attempts to keep the network subgraph synced.

https://github.com/graphprotocol/indexer/blob/f92c2f7587a67000821c188835f87c837a61315b/packages/indexer-agent/src/agent.ts#L993-L1003

I'll investigate why this isn't working as intended.

tilacog commented 1 year ago

Have any of you happened to update the subgraph network deployment identifier after starting the Agent?

I'm just checking to eliminate the hypothesis of a new subgraph deployment replacing an older one.

aasseman commented 1 year ago

Here's my chain of events:

Update from v0.20.16 to v0.20.20 (no settings changes)
Network deployment got removed
Thinking it's because mine is deprecated, updating the network deployment to QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs
Deployment gets added, then immediately removed too
Updating from v0.20.21
Deployment still gets removed

sbond14 commented 1 year ago

I followed the same process as @aasseman (except with an upgrade to v0.20.19 first).

After switching to the new network subgraph, I noticed it synced for a little while before being removed (I believe it caught all the way up).

Now after restarting my agent, these logs all occur within 6-7 seconds of each other: {"level":30,"time":1692986860134,"pid":1,"hostname":"indexer-agent-5687656845-ng8lt","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/5EFZk62JWs","deployment":{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"},"node_id":"index_node_0","msg":"Deploy subgraph deployment"} {"level":30,"time":1692986861097,"pid":1,"hostname":"indexer-agent-5687656845-ng8lt","name":"IndexerAgent","component":"GraphNode","name":"indexer-agent/5EFZk62JWs","deployment":{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"},"msg":"Successfully deployed subgraph deployment"} {"level":30,"time":1692986861097,"pid":1,"hostname":"indexer-agent-5687656845-ng8lt","name":"IndexerAgent","component":"GraphNode","deployment":{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"},"node":"index_node_0","msg":"Reassign subgraph deployment"} {"level":20,"time":1692986866475,"pid":1,"hostname":"indexer-agent-5687656845-ng8lt","name":"IndexerAgent","component":"Agent","syncing":[{"bytes32":"0x0fca528d7ca4f1f3bd8f07741d4d249d50abb2a6f4a7d8d3926db7f5d0464d7b","ipfsHash":"QmPQHm2yNmH1yRxd3Y2wXUzhXJTQv24GXigBQSvCBjmP2i"},{"bytes32":"0x5b83184ead3290c43e0a5524f086595e3ced90af300fbdfcfc07c7b884ebec54","ipfsHash":"QmUVskWrz1ZiQZ76AtyhcfFDEH1ELnRpoyEhVL8p6NFTbR"},{"bytes32":"0x06e9f2d25aa1c0e6063e663f46544ea4ea18e4697b54ba1f7eb2be94b91558b8","ipfsHash":"QmNoe4VQFSKAC3uiq48UQASg4QeqFVSKYYGwnxNjNi6GYX"},{"bytes32":"0xbe1876c22b0e5cfe6815342ede1c1817f324c79d1b5f6527b8ba7b1d7040c693","ipfsHash":"Qmb8hopjamZfxhSUf2UYiVAVnkBCHptrQn3oDqthh6Ly9U"},{"bytes32":"0x94f1b1fe3ba6f10529ccd9da8afe84986e142b1df01b315c61c131f69eeccf9d","ipfsHash":"QmYN4ofRb5CUg1WdpLhhNTVCuiiAt29hBKGjTnnxYh9zYt"},{"bytes32":"0xe3c1927e36be948ce1b52139c65ea4fd0af2656ca9dc75a89c899a00e7a48a10","ipfsHash":"QmdfiTEp7xUTuWqUS415aHYtSSzkVc4wT2BkgpwyCBwcF9"},{"bytes32":"0xd7fb49137a23fcee30010e79422660cf227481677f92d229421b473babc896e6","ipfsHash":"QmcskbugoHafPJC4EcFzkPNc2ExNPY8L32QagCn5qrtQU5"},{"bytes32":"0x9e32cc46ed3833723adfa867dac3196366e3ce87faa07323deab9a62cd56ae1b","ipfsHash":"QmYzC3xMfwpUNzycgcU7onNUPMF5UvChPrYwMiSitwEKRU"},{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"}],"target":[{"bytes32":"0xd7fb49137a23fcee30010e79422660cf227481677f92d229421b473babc896e6","ipfsHash":"QmcskbugoHafPJC4EcFzkPNc2ExNPY8L32QagCn5qrtQU5"},{"bytes32":"0xe3c1927e36be948ce1b52139c65ea4fd0af2656ca9dc75a89c899a00e7a48a10","ipfsHash":"QmdfiTEp7xUTuWqUS415aHYtSSzkVc4wT2BkgpwyCBwcF9"},{"bytes32":"0xbe1876c22b0e5cfe6815342ede1c1817f324c79d1b5f6527b8ba7b1d7040c693","ipfsHash":"Qmb8hopjamZfxhSUf2UYiVAVnkBCHptrQn3oDqthh6Ly9U"},{"bytes32":"0x06e9f2d25aa1c0e6063e663f46544ea4ea18e4697b54ba1f7eb2be94b91558b8","ipfsHash":"QmNoe4VQFSKAC3uiq48UQASg4QeqFVSKYYGwnxNjNi6GYX"},{"bytes32":"0x0fca528d7ca4f1f3bd8f07741d4d249d50abb2a6f4a7d8d3926db7f5d0464d7b","ipfsHash":"QmPQHm2yNmH1yRxd3Y2wXUzhXJTQv24GXigBQSvCBjmP2i"},{"bytes32":"0x5b83184ead3290c43e0a5524f086595e3ced90af300fbdfcfc07c7b884ebec54","ipfsHash":"QmUVskWrz1ZiQZ76AtyhcfFDEH1ELnRpoyEhVL8p6NFTbR"},{"bytes32":"0x94f1b1fe3ba6f10529ccd9da8afe84986e142b1df01b315c61c131f69eeccf9d","ipfsHash":"QmYN4ofRb5CUg1WdpLhhNTVCuiiAt29hBKGjTnnxYh9zYt"},{"bytes32":"0x9e32cc46ed3833723adfa867dac3196366e3ce87faa07323deab9a62cd56ae1b","ipfsHash":"QmYzC3xMfwpUNzycgcU7onNUPMF5UvChPrYwMiSitwEKRU"}],"withActiveOrRecentlyClosedAllocation":[{"bytes32":"0xe3c1927e36be948ce1b52139c65ea4fd0af2656ca9dc75a89c899a00e7a48a10","ipfsHash":"QmdfiTEp7xUTuWqUS415aHYtSSzkVc4wT2BkgpwyCBwcF9"},{"bytes32":"0xd7fb49137a23fcee30010e79422660cf227481677f92d229421b473babc896e6","ipfsHash":"QmcskbugoHafPJC4EcFzkPNc2ExNPY8L32QagCn5qrtQU5"}],"msg":"Reconcile deployments"} {"level":30,"time":1692986866475,"pid":1,"hostname":"indexer-agent-5687656845-ng8lt","name":"IndexerAgent","component":"Agent","deploy":[],"remove":[{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"}],"msg":"Deployment changes"} {"level":30,"time":1692986866477,"pid":1,"hostname":"indexer-agent-5687656845-ng8lt","name":"IndexerAgent","component":"GraphNode","deployment":{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"},"msg":"Remove subgraph deployment"} {"level":30,"time":1692986866500,"pid":1,"hostname":"indexer-agent-5687656845-ng8lt","name":"IndexerAgent","component":"GraphNode","deployment":{"bytes32":"0x6915de0cec60cd9cc874b735c2e061b556e241c80e4b278a12c6443fef4c9b80","ipfsHash":"QmVQrrgeGGHEqRdjAByeLvnNnDMjdt85jZZB5EFZk62JWs"},"msg":"Successfully removed subgraph deployment"}

tilacog commented 1 year ago

Fixed by #753

graphprotocol / indexer

Network subgraph gets removed #752