search

graphprotocol / graph-node

Graph Node indexes data from blockchains such as Ethereum and serves it over GraphQL
https://thegraph.com
Apache License 2.0
2.91k stars 978 forks source link

block not found in `triggers_by_block` error on Transfer event on Ropsten, block: 11026800 #3105

Closed cezary-stroczynski closed 2 years ago

cezary-stroczynski commented 2 years ago

In short: when I deploy THIS simple code I got the error:

Block stream produced a non-fatal error, error: block Block { hash: Some(0x2980930f65d4f67fecc807233e5cf62f146ce11e50bf54466daef9b1a577e3e4), parent_hash: 0x50e4b5405130bf8f31a3caa8e661b5128a1f808aa04a32247c90a5ecbe4db568, uncles_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, author: 0xca3f57dffbcf67c074b8cf54e4c873138facfc7f, state_root: 0x63d33c2b8f23a3553c71e633c38191c30e4f01820554d4bf36a7da47c91138db, transactions_root: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, receipts_root: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421, number: Some(11026800), gas_used: 0, gas_limit: 20000000, base_fee_per_gas: Some(7), extra_data: Bytes([98, 116, 114, 99, 45, 48, 49]), logs_bloom: Some(0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000), timestamp: 1631442575, difficulty: 6812130516, total_difficulty: Some(34827963848514944), seal_fields: [], uncles: [], transactions: [], size: Some(525), mix_hash: Some(0x4dea1ee4d29cbc777dadf5369f10908c069c1c08d6edbd33283876c3f5c99e33), nonce: Some(0x64d44a8aef0417f6) } not found in triggers_by_block, sgd: 1, subgraph_id: QmVC73TRGjNjr6w5w9aqqsd6oF5gofJHg9PYo5rqjgxi1a, component: SubgraphInstanceManager

It always occurs on the same block: 11026800 on Ropsten You can see the error in live version HERE (needs to extend log range to include Debug).

Current behaviour: In the provided code, there is a handler (handleAgreementTransfer) for standard ERC20 Transfer event. There is no provided contract address in subgraph.yaml, so the handler save all the Transfer events it encounter. Subgraph is saving all the Transfer events properly till the block 11026800, when the error occurs. It always happens on the same block: 11026800. I tested this behaviour on hosted service and on local graph-node. Both are giving the same result. After errror Subgraph stoped indexing any new blocks. The error is probably produced from THIS place in the code.

Steps to reproduce:

  1. Clone THIS repository
  2. yarn install and yarn codegen
  3. Deploy to hosted service or local graph-node. If deploy to local graph-node, make sure to set environment variables: GRAPH_LOG to debug and ethereum to ropsten:<your RPC>
  4. See the error in logs shortly after deploy

Expected behavior: It should save all Transfer events till the current block and not produce any errors.

maoueh commented 2 years ago

I wasn't able to reproduce, tested twice and each time the block in question was past without problem:

...
Jan 04 16:54:57.238 INFO Done processing trigger, gas_used: 66828316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 6, address: 0x67c9…880c, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0xea9da04fefc2d523319c6dee7a5f39c5b5523fc0728a7efbb702b6f417cdc9e0, block_number: 11026798, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.244 INFO Applying 9 entity operation(s), block_hash: 0xea9da04fefc2d523319c6dee7a5f39c5b5523fc0728a7efbb702b6f417cdc9e0, block_number: 11026798, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.330 INFO Scanning blocks [11026799, 11026800], range_size: 2, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.330 DEBG Requesting logs for blocks [11026799, 11026800], event 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef, 0 contracts, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.400 DEBG Found 2 relevant block(s), sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.402 DEBG Requesting 2 block(s), sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.498 DEBG Processing 105 triggers, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.498 DEBG 52 candidate triggers in this block, block_hash: 0x6b090ecfb7f2ba08001a81a30a4eba7b28978055a9a6c068c37965c2972f4f62, block_number: 11026799, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.506 INFO Done processing trigger, gas_used: 66838316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 6, address: 0x722d…2180, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x6b090ecfb7f2ba08001a81a30a4eba7b28978055a9a6c068c37965c2972f4f62, block_number: 11026799, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.513 INFO Done processing trigger, gas_used: 66838316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 6, address: 0xdd46…816c, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x6b090ecfb7f2ba08001a81a30a4eba7b28978055a9a6c068c37965c2972f4f62, block_number: 11026799, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.518 INFO Applying 3 entity operation(s), block_hash: 0x6b090ecfb7f2ba08001a81a30a4eba7b28978055a9a6c068c37965c2972f4f62, block_number: 11026799, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.535 DEBG 53 candidate triggers in this block, block_hash: 0x5b7f304f328fdf98467508c626dc328e2beb76f260c826da6cba6a3f0f1b13e6, block_number: 11026800, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.543 INFO Done processing trigger, gas_used: 66838316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 6, address: 0x2bef…90d0, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x5b7f304f328fdf98467508c626dc328e2beb76f260c826da6cba6a3f0f1b13e6, block_number: 11026800, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.549 INFO Done processing trigger, gas_used: 66778316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 5, address: 0xfd61…5da8, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x5b7f304f328fdf98467508c626dc328e2beb76f260c826da6cba6a3f0f1b13e6, block_number: 11026800, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.556 INFO Done processing trigger, gas_used: 66768316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 6, address: 0xfd61…5da8, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x5b7f304f328fdf98467508c626dc328e2beb76f260c826da6cba6a3f0f1b13e6, block_number: 11026800, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.562 INFO Applying 4 entity operation(s), block_hash: 0x5b7f304f328fdf98467508c626dc328e2beb76f260c826da6cba6a3f0f1b13e6, block_number: 11026800, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.664 INFO Scanning blocks [11026801, 11026801], range_size: 1, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.664 DEBG Requesting logs for blocks [11026801, 11026801], event 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef, 0 contracts, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.734 DEBG Found 1 relevant block(s), sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.744 DEBG Requesting 1 block(s), sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.796 DEBG Processing 50 triggers, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.797 DEBG 50 candidate triggers in this block, block_hash: 0x629325f5b90a2aac806d7d2ff56af77670e93129bfed2f22f88f939ef276ab91, block_number: 11026801, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:57.860 INFO Scanning blocks [11026802, 11026803], range_size: 2, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.861 DEBG Requesting logs for blocks [11026802, 11026803], event 0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef, 0 contracts, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.939 DEBG Found 2 relevant block(s), sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:57.940 DEBG Requesting 2 block(s), sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:58.024 DEBG Processing 107 triggers, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: BlockStream
Jan 04 16:54:58.025 DEBG 56 candidate triggers in this block, block_hash: 0x1fdf805790d40fad66cdb61eb09905b5757446230a06fac0bb459f4b4a6953a0, block_number: 11026802, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:58.040 INFO Done processing trigger, gas_used: 66778316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 7, address: 0xc377…d114, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x1fdf805790d40fad66cdb61eb09905b5757446230a06fac0bb459f4b4a6953a0, block_number: 11026802, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:58.047 INFO Done processing trigger, gas_used: 66828316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 6, address: 0xc778…d5ab, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x1fdf805790d40fad66cdb61eb09905b5757446230a06fac0bb459f4b4a6953a0, block_number: 11026802, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:58.054 INFO Done processing trigger, gas_used: 66838316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 6, address: 0x4552…d245, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x1fdf805790d40fad66cdb61eb09905b5757446230a06fac0bb459f4b4a6953a0, block_number: 11026802, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:58.060 INFO Done processing trigger, gas_used: 66788316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 6, address: 0x86f8…c401, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x1fdf805790d40fad66cdb61eb09905b5757446230a06fac0bb459f4b4a6953a0, block_number: 11026802, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:58.067 INFO Done processing trigger, gas_used: 66788316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 6, address: 0xdbc9…fc4e, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x1fdf805790d40fad66cdb61eb09905b5757446230a06fac0bb459f4b4a6953a0, block_number: 11026802, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:58.073 INFO Done processing trigger, gas_used: 66778316, data_source: Agreement, handler: handleAgreementTransfer, total_ms: 6, address: 0xc377…d114, signature: Transfer(indexed address,indexed address,uint256), block_hash: 0x1fdf805790d40fad66cdb61eb09905b5757446230a06fac0bb459f4b4a6953a0, block_number: 11026802, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
Jan 04 16:54:58.080 INFO Applying 7 entity operation(s), block_hash: 0x1fdf805790d40fad66cdb61eb09905b5757446230a06fac0bb459f4b4a6953a0, block_number: 11026802, sgd: 1, subgraph_id: QmZL8oBTgaUbjN6KmdDQnCquVEtVpWtoMjnsbsZvPSQhp7, component: SubgraphInstanceManager
...

Maybe it's fixed in master branch? Can you try with latest version of the code?

maoueh commented 2 years ago

Closing issue, feel free to re-open with more details if it's still a problem.

fortunaphantom commented 2 years ago

I am having this issue in Rinkeby. Block stream produced a non-fatal error, error: block #11289155 (45bfc1bf0f7e99448707d088c5258157cd403274773686493e7342747b73f075) not found intriggers_by_block`` @maoueh , @cezary-stroczynski Can you please help me?

0fprod commented 1 year ago

@fortunaphantom I'm facing a similar error: Block stream produced a non-fatal error, error: Failed to obtain triggers for block 8515355

Did you fix yours?

nekooei commented 10 months ago

I am having this issue in Rinkeby. Block stream produced a non-fatal error, error: block #11289155 (45bfc1bf0f7e99448707d088c5258157cd403274773686493e7342747b73f075) not found intriggers_by_block`` @maoueh , @cezary-stroczynski Can you please help me?

I have the same issue did you fix it?

maoueh commented 10 months ago

@nekooei Can you open a new issue with the needed details like the complete logs, will try to take a closer look.

ArseniiPetrovich commented 1 month ago

Just FYI - I found what the issue is. Basically if for some reason your node contains a fork and returns removed:true block in eth_getLogs request - The Graph Index node will not interpret this and make subgraph stuck. Probably, this is an expected behavior: https://github.com/graphprotocol/graph-node/blob/master/chain/ethereum/src/ethereum_adapter.rs#L1708C1-L1716C22 In these lines of code authors wrote that this function will not properly handle if the chain is a subject of reorgs.

That just caused with me.