Graph node throws error when procceding subgraph parsing.

[deathwing00000]

Instructions

Please fill out the template below to the best of your ability and include a label indicating which tool/service you were working with when you encountered the problem.

Problem

After deployment of the Graph node and deployment of the subgraph in the process block scanning this error occurs:

indexer-1      | Sep 04 04:49:03.570 ERRO Ethereum node returned an error when calling function "name" of contract "ERC20": RPC error: Error { code: ServerError(-32000), message: "failed to execute script at height: 213497601, with: invalid height not in available range: 213497601", data: None }, sgd: 1, subgraph_id: QmTNrQPkZv16zv5uL7cMhXsabUjfKpGQW346QXmMkKMHaJ, component: SubgraphInstanceManager
indexer-1      | Sep 04 04:49:03.570 INFO Done processing trigger, gas_used: 0, data_source: MorphoBlue, handler: handleCreateMarket, total_ms: 73555, transaction: 0x5cf8…9290, address: 0x5d7d…4bae, signature: CreateMarket(indexed bytes32,(address,address,address,address,uint256)), sgd: 1, subgraph_id: QmTNrQPkZv16zv5uL7cMhXsabUjfKpGQW346QXmMkKMHaJ, component: SubgraphInstanceManager
indexer-1      | Sep 04 04:49:03.570 INFO Possible reorg detected, retrying, error: failed to process trigger: block #2320931 (0x55b9…94eb), transaction 5cf88d62102ad61d932fa83b5091d654ac4c84b8105b615c03059af426279290: error while executing at wasm backtrace:
indexer-1      |     0: 0x5fa5 - <unknown>!~lib/@graphprotocol/graph-ts/chain/ethereum/ethereum.SmartContract#tryCall
indexer-1      |     1: 0x6397 - <unknown>!src/sdk/token/TokenManager#constructor
indexer-1      |     2: 0xf35c - <unknown>!src/initializers/markets/createMarket
indexer-1      |     3: 0x103b7 - <unknown>!src/morpho-blue/handleCreateMarket: Ethereum node returned an error when calling function "name" of contract "ERC20": RPC error: Error { code: ServerError(-32000), message: "failed to execute script at height: 213497601, with: invalid height not in available range: 213497601", data: None }, block_hash: 0x55b9ae92fe950cd29c485a4b3e57bcc0008ef523f711396e07e9bb6e3f1b94eb, block_number: 2320931, sgd: 1, subgraph_id: QmTNrQPkZv16zv5uL7cMhXsabUjfKpGQW346QXmMkKMHaJ, component: SubgraphInstanceManager

Basically subgraph met CreateMarket event and tried to execute handleCreateMarket function, which tried to fetch name from ERC20 contract and failed.

Another example:

Pretty the same flow, but with different SC.

Context

We are currently working on front end for the MoreMarkets Protocol and it highly depends on the subgraph and the data it returns. Currently it is the blocker for us and we will appreciate if You could look through it ASAP.

[sideninja]

The problem you are experiencing above is related to executing data at historic height which is no longer available on testnet. The problem is currently we don't keep a complete history for executing calls, so my question is why do you try to execute the call at historic height? FYI we are working on a feature to keep complete history but will take some time. On mainnet this shouldn't be an issue tho.

[deathwing00000]

so my question is why do you try to execute the call at historic height?

It is executed that way because of Graph node itself works with historic heights and we just tried to use Graph Node and their subgraph as the tool to fetch, analyze and store data that we can get from blockchain via parsing the blocks. Also Subgraphs are creating some sort of charts, so maybe for this it used execution of some method on historic heights.

[mmsys34]

And occasionally, eth_getBlockByHash returns null when I utilize the testnet public RPC URL.

I have tried this: curl --location 'https://testnet.evm.nodes.onflow.org' \ --header 'Content-Type: application/json' \ --data '{ "jsonrpc":"2.0", "method":"eth_getBlockByHash", "params":[ "0x2584c7ebd7710982d72988842935ef931c589fc81d0c598e39dd67b4d52b32e4", true ], "id":1 }

And the result is null: { "jsonrpc": "2.0", "id": 1, "result": null }

[sher9n]

It's very critical for us to resolve this asap given the launch date of the protocol on testnet. @sideninja and @m-Peter do we have a timeline on how soon this can be fixed?

[franklywatson]

We're investigating an issue with block processing which came up since the upgrade which is affecting the EVM side also. Please stay tuned on messaging/discord for updates, or subscribe here: https://status.onflow.org/

[sideninja]

@mmsys34 the get block issue was resolved but we are currently reindexing the network which takes some time, I believe today we should deploy EVM GW with that block issue fix.

About the historic execution, I will take a look whether we can support pre-network-upgrade heights. Will get back to you today.

[sideninja]

@mmsys34 can you give me more details about the request so I can try to reproduce, so what is the contract address, contract source and abi and what is the call you are doing?

[sideninja]

cc @sher9n @deathwing00000

[sideninja]

I assume this is the contract? https://evm-testnet.flowscan.io/address/0x5D7de68283A0AFcd5A1411596577CC389CDF4BAE?tab=txs

[mmsys34]

@mmsys34 can you give me more details about the request so I can try to reproduce, so what is the contract address, contract source and abi and what is the call you are doing?

We are monitoring 2 contracts: 1st: https://evm-testnet.flowscan.io/address/0x5D7de68283A0AFcd5A1411596577CC389CDF4BAE?tab=contract 2nd: https://evm-testnet.flowscan.io/address/0xb21fa39418F3AbBAE407124896B84980E7149b3e?tab=contract

I am operating the graph node on AWS with the Flow EVM testnet public RPC URL. However, during the block synchronization process, it gets stuck at the errors I have mentioned.

I reviewed the graph node logs, found the block number, manually attempted a cURL request, and received a response of null.

[deathwing00000]

I assume this is the contract? https://evm-testnet.flowscan.io/address/0x5D7de68283A0AFcd5A1411596577CC389CDF4BAE?tab=txs

@sideninja

Basically in the transaction, that was mentioned in the error, CreateMarket() event was emitted by 0x5D7de68283A0AFcd5A1411596577CC389CDF4BAE.

Subgraph parsed that event and tried to do basic script, part of which is get the name of the ERC20 contract, that was provided in the params of event.

I believe it stucked on execution of view method name() on the ERC20 contract with address 0xa2ee7435820cd09c35e54ba19361c35fc423fe23 or 0xfd1a1dc5567519b3f4225268cb6bf943bf8cd9c2

[sideninja]

Thank you... looking into it with @m-Peter

[m-Peter]

curl -s -XPOST 'https://testnet.evm.nodes.onflow.org' --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","id":1,"method":"eth_call","params":[{"from":"0x2690eef879df58b97ec7cf830f7627fb1b51f826","gas":"0x0","gasUsed":"0x79b78","input":"0x8c1358a2000000000000000000000000a2ee7435820cd09c35e54ba19361c35fc423fe23000000000000000000000000fd1a1dc5567519b3f4225268cb6bf943bf8cd9c2000000000000000000000000d7d48a64eb6f2ddf740d3258d2df7b2e7122f01a000000000000000000000000cfc5f61bff3fec063a73038e97bbe883360547e70000000000000000000000000000000000000000000000000b1a2bc2ec500000","to":"0x5d7de68283a0afcd5a1411596577cc389cdf4bae","value":"0x0"},"0x236A23"]}'

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32000,
    "message": "failed to execute script at height: 213497601, with: invalid height not in available range: 213497601"
  }
}

I was able to find the exact call that produces this error. Now we just have to debug why that happens. Transaction can be found at: https://evm-testnet.flowscan.io/tx/0x5cf88d62102ad61d932fa83b5091d654ac4c84b8105b615c03059af426279290?tab=raw_trace

[m-Peter]

ERR:  client: rpc error: code = InvalidArgument desc = failed to execute the script on the execution node
execution-002.devnet51.nodes.onflow.org:3569: rpc error: code = InvalidArgument desc = failed to execute script:
failed to create storage snapshot: state not found in ledger for commit
33c3f33d427507544e8d71aeef3e6df3ec6d1330617df73249570fdb5a2d245f (block 
2c770c4638641c247191814542e3e6615bce9075c752177aba240f20f6221944): execution state is pruned

This is the exact error message ^

[mmsys34]

This is the exact error message ^

This error is related with our smart contracts or subgraph code?

[m-Peter]

This is the exact error message ^

This error is related with our smart contracts or subgraph code?

Not at all, it has to do with our setup, so worry not.

[sideninja]

We are experiencing an issue where the script is executed in the wrong place. We are looking into as to why and will get back asap.

[sher9n]

HI @sideninja, do we have an update on this?

[sideninja]

@sher9n @mmsys34 the blocks should be now available. The call failing will likely need reconfiguring of AN to try again. @peterargue can we increase the compute limit on the AN access-002.devnet51 to see if this will allow the execution on the AN. What is weird tho, is that this request shouldn't be computationally intensive so not sure why it gets forwarded to EN instead of AN. This is similar to what we see on mainnet, but for this case they just read name of a contract. We will try to figure out how much compute it does use to confirm this. But can we try increase the limit to see if this is resolved.

[deathwing00000]

@sher9n @mmsys34 the blocks should be now available. The call failing will likely need reconfiguring of AN to try again. @peterargue can we increase the compute limit on the AN access-002.devnet51 to see if this will allow the execution on the AN. What is weird tho, is that this request shouldn't be computationally intensive so not sure why it gets forwarded to EN instead of AN. This is similar to what we see on mainnet, but for this case they just read name of a contract. We will try to figure out how much compute it does use to confirm this. But can we try increase the limit to see if this is resolved.

Still meeting pretty the same issue on the same height:

indexer-1      | Sep 06 12:29:44.895 INFO Syncing 5 blocks from Ethereum, code: BlockIngestionStatus, blocks_needed: 5, blocks_behind: 5, latest_block_head: 3912349, current_block_head: 3912344, provider: testnet-rpc-0, component: EthereumPollingBlockIngestor
indexer-1      | Sep 06 12:29:46.133 WARN Trying again after eth_call RPC call for block #2320931 (55b9ae92fe950cd29c485a4b3e57bcc0008ef523f711396e07e9bb6e3f1b94eb) failed (attempt #10) with result Err(Web3Error(Rpc(Error { code: ServerError(-32000), message: "failed to execute script at height: 213497601, with: invalid height not in available range: 213497601", data: None }))), provider: testnet-rpc-0, sgd: 1, subgraph_id: QmTNrQPkZv16zv5uL7cMhXsabUjfKpGQW346QXmMkKMHaJ, component: SubgraphInstanceManager
indexer-1      | Sep 06 12:29:46.133 ERRO Ethereum node returned an error when calling function "name" of contract "ERC20": RPC error: Error { code: ServerError(-32000), message: "failed to execute script at height: 213497601, with: invalid height not in available range: 213497601", data: None }, sgd: 1, subgraph_id: QmTNrQPkZv16zv5uL7cMhXsabUjfKpGQW346QXmMkKMHaJ, component: SubgraphInstanceManager
indexer-1      | Sep 06 12:29:46.133 INFO Done processing trigger, gas_used: 0, data_source: MorphoBlue, handler: handleCreateMarket, total_ms: 87150, transaction: 0x5cf8…9290, address: 0x5d7d…4bae, signature: CreateMarket(indexed bytes32,(address,address,address,address,uint256)), sgd: 1, subgraph_id: QmTNrQPkZv16zv5uL7cMhXsabUjfKpGQW346QXmMkKMHaJ, component: SubgraphInstanceManager
indexer-1      | Sep 06 12:29:46.134 INFO Possible reorg detected, retrying, error: failed to process trigger: block #2320931 (0x55b9…94eb), transaction 5cf88d62102ad61d932fa83b5091d654ac4c84b8105b615c03059af426279290: error while executing at wasm backtrace:
indexer-1      |     0: 0x5fa5 - <unknown>!~lib/@graphprotocol/graph-ts/chain/ethereum/ethereum.SmartContract#tryCall
indexer-1      |     1: 0x6397 - <unknown>!src/sdk/token/TokenManager#constructor
indexer-1      |     2: 0xf35c - <unknown>!src/initializers/markets/createMarket
indexer-1      |     3: 0x103b7 - <unknown>!src/morpho-blue/handleCreateMarket: Ethereum node returned an error when calling function "name" of contract "ERC20": RPC error: Error { code: ServerError(-32000), message: "failed to execute script at height: 213497601, with: invalid height not in available range: 213497601", data: None }, block_hash: 0x55b9ae92fe950cd29c485a4b3e57bcc0008ef523f711396e07e9bb6e3f1b94eb, block_number: 2320931, sgd: 1, subgraph_id: QmTNrQPkZv16zv5uL7cMhXsabUjfKpGQW346QXmMkKMHaJ, component: SubgraphInstanceManager

[sideninja]

@deathwing00000 yes the issue should still be there, we will try to increase limits soon now. I will ping you.

[sideninja]

@deathwing00000 can you try again?

[deathwing00000]

@sideninja Still getting the same error.

[mmsys34]

@sideninja I have tested on my local docker container, and got same error.

[deathwing00000]

@sideninja Is there any update on this?

[sher9n]

@sideninja and @m-Peter, can you guys please investigate asap. We're aiming for a testnet launch on Friday and will have to delay if this is not resolved. I would really like to avoid that given the FLOW commits.

[m-Peter]

Hey @sher9n, I am afraid that on my side I cannot do much else. I found the root cause of the problem, but I am not a member of the Flow team, hence I have no access to infra tools etc. I am just a community developer. I think that Gregor is out today, so you'll have to wait for his response on the matter :pray:

[peterargue]

I took a closer look. The height those scripts use is 213497601. This is before the last height-coordinated upgrade on testnet, which happened at 214288308.

Access nodes are only able to execute scripts against blocks that were produced with a compatible version of FVM. During HCUs, the FVM version is updated, so blocks from before the upgrade can no longer be executed. In that case, they forward the request to an execution node. ENs only have a small amount of history, so the request fails.

I know there is a project in the works that should solve this. @sideninja will have some more info when he's back.

[deathwing00000]

@peterargue @sideninja This script is actually using 213497601 because of the fact that it emitted an event in this block. If we re-deploy the smart contract and redo all the configurations, it should work fine for now. The question is, can we rely on the next HCU not to happen and break everything for us again.... Perhaps you have some opinion on this?

[sideninja]

We are pushing out a feature on testnet that will completely avoid relying on AN to execute any calls or transactions. This will avoid any issues that you are facing. I'm really sorry for all these troubles.

[deathwing00000]

@sideninja That sounds really great, thank You a lot! Maybe You have any estimations when it could go live?

[sher9n]

Hi @sideninja, do we have an ETA on when this would be completed? We really would like to get this going asap so we can launch on testnet.