goerli: The Epoch Subgraph is 2 epochs away from the Epoch Manager

[ThomasBlock]

Thank you for updating indexer-agent ( https://github.com/graphprotocol/indexer/issues/646 ). i can now run v0.20.16

now i am confronted with the next problem. i can not see the actual allocations and so can not (force)close them:

graph-indexer indexer allocations get                
✖ [GraphQL] Failed to query Epoch Block Oracle Subgraph

{"level":20,"time":1682011182036,"pid":107521,"hostname":"thegraph-goerli","name":"IndexerAgent","component":"IndexerManagementClient","filter":{"status":"active","allocation":null},"msg":"Execute allocations() query"}
{"level":50,"time":1682011182393,"pid":107521,"hostname":"thegraph-goerli","name":"IndexerAgent","component":"NetworkMonitor","epochManagerCurrentEpoch":2956,"epochSubgraphCurrentEpoch":"2954","epochDifferenceFromContracts":2,"networkID":"eip155:5","msg":"The Epoch Subgraph is 2 epochs away from the Epoch Manager. It should never be more than 1 epoch away from the Epoch Manager contract. Please submit an issue at https://github.com/graphprotocol/block-oracle/issues/new"}

i could also reallocate two fantom allocations today - so i guess there should be no problem on fantom or ethereum side ( i am with both on chainhead )

is there anything else i might have to update?

my updates:

npm install -g @graphprotocol/indexer-service@latest
npm install -g @graphprotocol/indexer-agent@latest
npm install -g @graphprotocol/indexer-cli@latest
npm install -g @graphprotocol/indexer-common@latest
npm install -g @graphprotocol/indexer-native@latest
npm install -g @graphprotocol/graph-cli@latest

my commandlines:

graph-indexer-service start \
    --network-subgraph-endpoint https://api.thegraph.com/subgraphs/name/graphprotocol/graph-network-goerli \
    --client-signer-address 0xe1EC4339019eC9628438F8755f847e3023e4ff9c \
    --network-subgraph-deployment QmPVz18RFwK6hE5rZFWERk23LgrTBz2FCkZzgPSrFxFWN4

graph-indexer-agent start \
    --network-subgraph-endpoint https://api.thegraph.com/subgraphs/name/graphprotocol/graph-network-goerli \
    --collect-receipts-endpoint https://gateway.testnet.thegraph.com/collect-receipts \
    --dai-contract 0x9e7e607afd22906f7da6f1ec8f432d6f244278be \
    --epoch-subgraph-endpoint https://api.thegraph.com/subgraphs/name/graphprotocol/goerli-epoch-block-oracle \
    --network-subgraph-deployment QmPVz18RFwK6hE5rZFWERk23LgrTBz2FCkZzgPSrFxFWN4 \

[ThomasBlock]

without doing changes and two days with errors, it now worked again. ( currentEpochNumber:2972 )

[begetan]

"err": "The Epoch Subgraph is 4 epochs away from the Epoch Manager. It should never be more than 1 epoch away from the Epoch Manager contract. Please submit an issue at https://github.com/graphprotocol/block-oracle/issues/new",
  "msg": "Epoch subgraph could not be queried"

[tilacog]

Thanks for the report.

We had an incident with Ethereum Goerli last week, indeed, but it is now solved and the Epoch Subgraph is in sync again.

Retroactively, we couldn't find any case were the block distance was that long on our mainnets.

I'm closing this for now, but feel free to reopen with more details or if this is happens again.

[ThomasBlock]

Thank you for the feedback. now happening again ( for optimism )

{level":50,"time":1682345786024,"pid":113300,"hostname":"thegraph-goerli","name":"IndexerAgent","component":"NetworkMonitor","err":{"type":"Error","message":"The Epoch Subgraph is 6 epochs away from the Epoch Manager. It should never be more than 1 epoch away from the Epoch Manager contract. Please submit an issue at https://github.com/graphprotocol/block-oracle/issues/new","stack":"Error: The Epoch Subgraph is 6 epochs away from the Epoch Manager. It should never be more than 1 epoch away from the Epoch Manager contract. Please submit an issue at https://github.com/graphprotocol/block-oracle/issues/new\n at NetworkMonitor.<anonymous> (/usr/lib/node_modules/@graphprotocol/indexer-agent/node_modules/@graphprotocol/indexer-common/dist/indexer-management/monitor.js:562:27)\n at Generator.next (<anonymous>)\n at fulfilled (/usr/lib/node_modules/@graphprotocol/indexer-agent/node_modules/@graphprotocol/indexer-common/dist/indexer-management/monitor.js:5:58)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)","attemptNumber":6,"retriesLeft":0},"msg":"Failed to query latest epoch number","networkID":"eip155:10","networkAlias":"optimism"}

[ThomasBlock]

i also see this error - maybe related?

{"level":40,"time":1682347799963,"pid":115530,"hostname":"thegraph-goerli","name":"IndexerAgent","component":"Agent","err":{"type":"IndexerError","message":"Failed to reconcile indexer and network","stack":"IndexerError: Failed to reconcile indexer and network\n at indexerError (/usr/lib/node_modules/@graphprotocol/indexer-agent/node_modules/@graphprotocol/indexer-common/dist/errors.js:171:12)\n at Agent.<anonymous> (/usr/lib/node_modules/@graphprotocol/indexer-agent/dist/agent.js:248:64)\n at Generator.throw (<anonymous>)\n at rejected (/usr/lib/node_modules/@graphprotocol/indexer-agent/dist/agent.js:6:65)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)","code":"IE005","explanation":"https://github.com/graphprotocol/indexer/blob/main/docs/errors.md#ie005","cause":{"type":"CombinedError","message":"[GraphQL] sorry, too many clients already","name":"CombinedError","graphQLErrors":[{"message":"sorry, too many clients already","path":["queueActions"]}]}},"msg":"Exited early while reconciling deployments/allocations"}

[grumas-rc]

[16:32:35.128] ERROR (IndexerAgent/471308): The Epoch Subgraph is 7 epochs away from the Epoch Manager. It should never be more than 1 epoch away from the Epoch Manager contract. Please submit an issue at https://github.com/graphprotocol/block-oracle/issues/new component: "NetworkMonitor" epochManagerCurrentEpoch: 2996 epochSubgraphCurrentEpoch: "2989" epochDifferenceFromContracts: 7 networkID: "eip155:5" [16:32:35.128] WARN (IndexerAgent/471308): Epoch subgraph could not be queried component: "NetworkMonitor" networkID: "eip155:5" networkAlias: "goerli" attempt: 1 retriesLeft: 5 err: "The Epoch Subgraph is 7 epochs away from the Epoch Manager. It should never be more than 1 epoch away from the Epoch Manager contract. Please submit an issue at https://github.com/graphprotocol/block-oracle/issues/new"

[tilacog]

Thanks for the report. Closing this as this incident is now resolved. Please feel free to reopen in case this happens again.