Local Graph-node not fully syncing large volume of entity updates

[0xTimepunk]

Do you want to request a feature or report a bug? Bug What is the current behavior? I have an issue where in a fork of aave v2 subgraph a subset of a certain entity type disappears from the database (or is never present to begin with).

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

The issue can be reproduced like this:

If I deploy the subgraph onto graph-node AFTER my smart contracts tests end, I get the issue where a subset of ids are not even present in the database (it's like they never existed in first place)

A work around for the solution is:

Deploying the subgraph AS SOON AS the contracts finish compiling and start to be deployed onto hardhat local evm (Before the tests start). This way the subgraph picks everything up correctly and there seem to be no deletion issues.

What is the expected behavior?

Whenever I deploy the subgraph onto the local graph-node it picks all entities correctly despite of the volume

Some notes and ideas:

Please note that in both cases I didn't change anything in the entities yaml file nor any mapping, the workaround is the only difference, so I am guessing some bug in graph-node.

Could it be because the local graph-node can't handle a large volume of entities being updated at the same time so it drops some of them? My tests involve a lot of entities changes - if I deploy the subgraph after the tests, everything tries to be synced at the same time.

I am on the latest graph-node master branch on github.

[azf20]

hi - can you provide a bit more detail on the setup you are using? Do you run the smart contract tests on the chain that you are then indexing on the subgraph?

[0xTimepunk]

I run the tests on the same local evm node than then I am indexing on the subgraph. These tests consist on an amount of deposits/borrows/repays being done in rapid succession (block after block) to test the subgraph state vs the on-chain state. Some mappings involve loops when updating entities which introduces some volume of changes over aave v2 subgraph

Smart contract side setup:

• hardhat local evm node (hardhat node) ["hardhat": "2.6.1"]
• smart contracts compiled with types produced by typechain ["typechain": "^5.1.2"]

hardhat config setup for the network I'm using:

  hardhatevm: {
    hardfork: 'berlin',
    blockGasLimit: 9500000,
    gas: 9500000,
    gasPrice: 8000000000,
    chainId: 31337,
    throwOnTransactionFailures: true,
    throwOnCallFailures: true,
    url: 'http://localhost:8545',
  },

Subgraph setup:

• Docker desktop + WSL2
• indexing from block 0

Others: [ "@graphprotocol/graph-cli": "^0.21.1"] ["@graphprotocol/graph-ts": "^0.20.0"]

Again, the issue only appears if I deploy the subgraph at the end of the test (which then tries to sync everything in rapid succession)

[azf20]

Subgraphs shouldn't have a problem with syncing lots of data - are you sure that the testing hardhat EVM isn't ephemeral? Can you verify that the events are present on-chain? (e.g. with Ethers event filtering)

[0xTimepunk]

Subgraphs shouldn't have a problem with syncing lots of data - are you sure that the testing hardhat EVM isn't ephemeral? Can you verify that the events are present on-chain? (e.g. with Ethers event filtering)

@azf20 I am in contact at a #spectral-finance chat with Simon regarding this issue.

Btw, the EVM isn't ephemeral and events are present on chain

[SkandaBhat]

Any updates on this issue? I am facing the same exact problem. @azf20 @0xTimepunk