[Bug] graph-node temporary outage with `store error: NotFound`

[SozinM]

Bug report

We experience an error (logs attached) while running the subgraph causing a temporary sync stop:

Seems like the error is linked to some store problem but we can't find any problems. All metrics are good for both PostgreSQL and graph-node.

99% of the time node is working absolutely fine. Also, this problem occurs on all subgraphs at the same time.

Graph version is v0.31.0 Docker image is graphprotocol/graph-node:v0.31.0

Relevant log output

Jul 04 08:23:51.258 DEBG Block stream produced a non-fatal error, error: store error: NotFound, sgd: 319, subgraph_id: QmQy5znQNDrZiVq8L7o3zFaTadiQz3tzgfXASBLz2WFgXL, component: SubgraphInstanceManager

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out

• [ ] Tick this box if this bug is caused by a regression found in the latest release.
• [ ] Tick this box if this bug is specific to the hosted service.
• [X] I have searched the issue tracker to make sure this issue is not a duplicate.

OS information

None

[SozinM]

Also so far I found that cache hist misses correlate with the moment of temporary outage:

[SozinM]

It seems like the reason for this behavior is that sometimes Graph starts to write a LOT of transactions to PostgreSQL NOTE that this picture does not correlate with the picture above. It's different days.

[SozinM]

Also from the source code it seems that error captured is: StoreError::Unknown(DieselError::NotFound)

[azf20]

hey @SozinM what version of Graph Node are you running?

[SozinM]

Hi @azf20 ! Sorry for missing that. We are using docker image graphprotocol/graph-node:v0.31.0 Which corresponds to the latest released version.

[azf20]

Thanks - how many subgraphs are you running on your Graph Node?

[SozinM]

I think it's about 50 on this one

[SozinM]

@azf20 any ideas?

[azf20]

cc @lutter the intermittent store issue, in case this helps

[PekopT]

@lutter @azf20 bump please, we're struggling with reliability

[azf20]

I wonder if there is something specific with the subgraphs or networks you are running which is causing this (particularly given the spikes in transactions)? Do you see this on all instances or just in one case?

[SozinM]

We see it on multiple indexers and I assume it is directly linked to the load on Postgres. We fist saw this behavior on loaded indexers. Also, we store block cache with subgraph on the single Postgres indexer.

[lutter]

NotFound indicates that graph-node tried to look up a row that does not exist; it shouldn't be dependent on load. Can you check your postgres logs for any errors?

[SozinM]

I was monitoring Postgresql and did not see any anomalies. The only errors I saw - were about connection dropping because the consumer does not respond (or something along these lines). @PekopT @balakhonoff check database again please