search

graphprotocol / mission-control-indexer

Technical indexer documentation and infrastructure templates for the Mission Control testnet
21 stars 4 forks source link

Indexing fatal errors - document them here only #116

Closed cryptovestor21 closed 3 years ago

cryptovestor21 commented 3 years ago

@fattox now tracking this for P2 https://github.com/graphprotocol/mission-control-indexer/issues/169, closing this issue.

If you have a fatal indexing error that is not already documented here please add it. Before adding an issue, please restart your graph nodes as this will sometimes clear non-fatal indexing errors.

Known subgraph issues:

Name ipfshash State Comments
Compound QmdYA3zCs27962HNchhcjJbyfT3hZtLhANJuo1BsMv5YTs Ongoing Multiple reports
Curve QmNs5fU6VkyGUsaNsa5sUHEKpzCA5AZWKJG8LaBrcCRpmW Ongoing Multiple reports
Uniswap V2 QmWTrJJ9W8h3JE19FhCzzPYsJ2tgXZCdUqnbyuo64ToTBN Ongoing Multiple reports
Uniswap V1 QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP Ongoing Multiple reports
Known Origin QmUxpVrcmXRmJgi8CMXu8JPHABspnoaewQjDujc17bkGqR Ongoing Multiple reports (some fix with restart others do not

Edit: lots of productive discussion about fatal errors in general with the team over last few days. See these discussions:

Zac: https://discordapp.com/channels/438038660412342282/755239678566858773/762017874142429205 https://discordapp.com/channels/438038660412342282/755239678566858773/761601730424471612

Dave: https://discordapp.com/channels/438038660412342282/755239678566858773/762351295452413974

Yaniv: https://discordapp.com/channels/438038660412342282/755239678566858773/762040425157623808

Jannis: https://discordapp.com/channels/438038660412342282/755239678566858773/761595531809914890 https://discordapp.com/channels/438038660412342282/755239678566858773/761601730424471612

Curve fatal indexing error:

Block 10950815 Subgraphid QmNs5fU6VkyGUsaNsa5sUHEKpzCA5AZWKJG8LaBrcCRpmW (Curve) Message Failed to process trigger in block #10950815 (33821d480038c26acbc85fdae5660d550581d607016f03eb8b2bb2dc579b38db), transaction de1d5953cb6f144d8e57379042c81c336ab95ed8efb23e5adca4e6b0df986286: function handleRampA not found

graph-node updated to 

 * branch              master     -> FETCH_HEAD
   b966adfc..18b61832  master     -> origin/master
Updating b966adfc..18b61832

Eth node OE3.x

restarting graph or eth nodes has no impact on the behaviour.

Jannis commented 3 years ago

@cryptovestor#5714 That sounds like a permanent bug in the subgraph that will require a subgraph update to fix. I'll check with the team.

Wanderer-Cry commented 3 years ago

confirmed, had the same today and reproduced all as well

subraph stopped syncing

OE node is 3.x

      {
        "chains": [
          {
            "chainHeadBlock": {
              "number": "10951237"
            },
            "latestBlock": {
              "number": "10950814"
            }
          }
        ],
        "fatalError": {
          "message": "Failed to process trigger in block #10950815 (33821d480038c26acbc85fdae5660d550581d607016f03eb8b2bb2dc579b38db), transaction de1d5953cb6f144d8e57379042c81c336ab95ed8efb23e5adca4e6b0df986286: function handleRampA not found"
        },
        "subgraph": "QmNs5fU6VkyGUsaNsa5sUHEKpzCA5AZWKJG8LaBrcCRpmW",
        "synced": true
      },
koen84 commented 3 years ago

For me this subgraph keeps indexing fine. Currently on block 10960857

cryptovestor21 commented 3 years ago

I've modified this issue to be a catch-all for all fatal indexing errors to try and reduce noise.

cryptovestor21 commented 3 years ago

Uni v1 fatal error:

QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP

Failed to process trigger in block #10973257 (ef979404a206a6943997e46062dac14d4d529715180654148fb862bcacaf5a06), transaction 75d68bb67bc700c19d531b807703916dc4e49182e1399b72b3913b0f21061dab: Failed to invoke handler 'handleSync': attempted to divide BigDecimal `123` by zero wasm backtrace: 0: 0x16ce - <unknown>!<wasm function 68> 1: 0x2809 - <unknown>!<wasm function 161>

koen84 commented 3 years ago

Confirming QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP

I tried subgraph_reassign and restarting graph-node, remains fatal error and not indexing.

koen84 commented 3 years ago

QmUxpVrcmXRmJgi8CMXu8JPHABspnoaewQjDujc17bkGqR (known origin) has been broken for a while too.

huglester commented 3 years ago 
        "fatalError": {
          "message": "Failed to process trigger in block #10950815 (33821d480038c26acbc85fdae5660d550581d607016f03eb8b2bb2dc579b38db), transaction de1d5953cb6f144d8e57379042c81c336ab95ed8efb23e5adca4e6b0df986286: function handleRampA not found"
        },
        "subgraph": "QmNs5fU6VkyGUsaNsa5sUHEKpzCA5AZWKJG8LaBrcCRpmW"

re-deployment and graph-node restart - no help.

cryptovestor21 commented 3 years ago

QmdYA3zCs27962HNchhcjJbyfT3hZtLhANJuo1BsMv5YTs (Compound) at 10991589

Failed to process trigger in block #10991589 (0159371824125a1e4ad22066202c910541afc991b8ef3b0a69e7c90c35d1df80), transaction f4e9f5e6dbc3637176c892c937a8053c9292e7d5051a61205744ee2a86cffc8d: Failed to invoke handler 'handleLiquidateBorrow': Mapping aborted at ~lib/@graphprotocol/graph-ts/index.ts, line 782, column 4, with message: Value is not a BigInt. wasm backtrace: 0: 0x13bd - !<wasm function 63> 1: 0x13d1 - !<wasm function 64> 2: 0x1876 - !<wasm function 77>

Does not resolve with node restart

Bambarello commented 3 years ago

So more and more subgraphs are broken, who's guilty? developers of subgraphs? Is there any workaround from Graph-Indexer code required to improve the situation with errors and blocked subgraph sync?

Bambarello commented 3 years ago

I have the same issue with "ZP", nothing helps, redeployment, restart, delete, re-deployment... no go.

        "chains": [
          {
            "chainHeadBlock": {
              "number": "10996521"
            },
            "latestBlock": {
              "number": "10973256"
            },
            "network": "mainnet"
          }
        ],
        "fatalError": {
          "message": "Failed to process trigger in block #10973257 (ef979404a206a6943997e46062dac14d4d529715180654148fb862bcacaf5a06), transaction 75d68bb67bc700c19d531b807703916dc4e49182e1399b72b3913b0f21061dab: Failed to invoke handler 'handleSync': attempted to divide BigDecimal `123` by zero\nwasm backtrace:\n  0: 0x16ce - <unknown>!<wasm function 68>\n  1: 0x2809 - <unknown>!<wasm function 161>\n"
        },
        "subgraph": "QmXKwSEMirgWVn41nRzkT3hpUBw29cp619Gx58XW6mPhZP",
        "synced": true
cryptovestor21 commented 3 years ago

So more and more subgraphs are broken, who's guilty? developers of subgraphs? Is there any workaround from Graph-Indexer code required to improve the situation with errors and blocked subgraph sync?

Check discord links in first post

koen84 commented 3 years ago

@cryptovestor21 would it be an idea you update the beginning of the first post with a list of confirmed subgraphs with issues ? Then this can be an easy point of reference for indexers to know which are problematic, respectively which might be something on their end.

cryptovestor21 commented 3 years ago

@cryptovestor21 would it be an idea you update the beginning of the first post with a list of confirmed subgraphs with issues ? Then this can be an easy point of reference for indexers to know which are problematic, respectively which might be something on their end.

done

koen84 commented 3 years ago

I'm seeing QmW6BrjSMPUSjMApTNYZShHi9XShgziC5YppGvqyjjAAX1 with fatal error ?

koen84 commented 3 years ago

These additional subgraphs seem to be failing in P2 : QmQrapV3GN6GnkMByUbf6Xrw1xVqSNAbQtNkXsy6xKAs15

EDIT : these seem fine meanwhile (could've been tracing subgraph on none tracing indexer) QmcPSRpF28QSWpgAgNPFVyiGYd75L5aSrKGLXhmaFW5bSK QmWex7boz79MGkYmcLbjDg4N13kLsLQAtTfLyjcz6RgJ9i QmXwR6LiGaP9yYTsSq8wGUS91Bqat7dStDsuH2XSoeKctY

Wanderer-Cry commented 3 years ago

One more issue in Phase 2:

QmfCaCcQnfBWXgCpbtf9kKWosVqUbeGLfEomYVJkMBqKj9

Failed to process trigger in block #11074853 (e1690ffc55669dc9b1bfcc9b5c8e75be176bb9938cdb3617cca1aeb63c81c942), transaction 89ed36656190f8f5cc16af8ee6665f2be64da289022cb0008e0fc4a6851fe07f: Failed to invoke handler 'handleVoted': Mapping aborted at generated/schema.ts, line 277, column 4, with message: Cannot save Dispute entity without an ID
wasm backtrace: 0: 0x16e6 - <unknown>!<wasm function 67> 1: 0x1882 - <unknown>!<wasm function 73>
trader-payne commented 3 years ago 

10053060 | QmaEsTQhqGfZMjHYpNxmcxU3pD4GbAU6u5F8AqePeQumYv | Failed to process trigger in block #10053060 (d2cfeb2082be49303054ce8e4b4e7d78e4bac8e0edcace2180921ea920850db3), transaction e4d33553e88cca3ebd09f4310865f82e3745ac19e03d15530bebd5566b78b71d: Failed to invoke handler 'updatePollTallyOnUnbond': Mapping aborted at ~lib/@graphprotocol/graph-ts/chain/ethereum.ts, line 439, column 6, with message: Call reverted, probably because an `assert` or `require` in the contract failed, consider using `try_pendingStake` to handle this in the mapping. wasm backtrace: 0: 0x159c - <unknown>!<wasm function 65> 1: 0x1605 - <unknown>!<wasm function 67> 2: 0x1dfe - <unknown>!<wasm function 96> 3: 0x1e62 - <unknown>!<wasm function 97>
-- | -- | --
koen84 commented 3 years ago

More fatal crashes : QmYPh4BHfJRszYBP7tnPFPHpKf2DdyHAtgRbE8PAnU1fZN QmXdZDKnDX2o9qfupGDTXFM1xxspndHrHAGpbkWAPMs6Qk QmW5UcMhXwXrMqRpTdT3MKB7HYDxbsp5oe9Atkuh4sgAA5 QmfCaCcQnfBWXgCpbtf9kKWosVqUbeGLfEomYVJkMBqKj9 Qmd8JSo29vQX3iP45JMVEL2zQJ6bQEMkqnWsoW3eRjMJEn

koen84 commented 3 years ago

And this one too now : QmWpFQJWVBkzTfabZL5DxnAvdhjP5FjbBW87jP2XcjyuUb

cryptovestor21 commented 3 years ago

@fattox now tracking this for P2 https://github.com/graphprotocol/mission-control-indexer/issues/169, closing this issue.