search

graphprotocol / graph-node

Graph Node indexes data from blockchains such as Ethereum and serves it over GraphQL
https://thegraph.com
Apache License 2.0
2.89k stars 962 forks source link

Resource exhausted `message larger than max (9418493 vs. 4194304)` #4319

Closed paymog closed 1 year ago

paymog commented 1 year ago

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

The graph node emits the following log line:

Jan 25 12:50:31.611 ERRO An error occurred while streaming blocks: status: ResourceExhausted, message: "grpc: received message larger than max (9418493 vs. 4194304)", details: [], metadata: MetadataMap { headers: {} }, sgd: 3112, subgraph_id: QmacUjeDurvnnfNnubtKxtVQve5iEnHs5B9bDAZbwC1prP, component: FirehoseBlockStream

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

Not sure how to repro but here's my set up:

  1. I'm running graph node 0.28.2
  2. I'm running firehose with image ghcr.io/streamingfast/firehose-ethereum:4378d28 (which has some bugfixes on top of ghcr.io/streamingfast/firehose-ethereum:develop-geth-v1.10.26-fh2.1)
  3. I have features = [ "filters" ] turned on for my firehose provider in the graph node toml

My index files were recently rebuilt and this issue only happens when features = [ "filters" ] is used by graph node. If I remove that feature from the firehose provider in my graph node toml, this issue goes away.

What is the expected behavior?

I expect graph node to correctly set the maximum size of the grpc messages it can accept.

paymog commented 1 year ago

It seems this only happens for a few of the subgraphs deployed into my infrastructure: QmWyLVicJp1zPpMAhzBEFEJ5qAES678J3oBFomUbpCrMyq, QmacUjeDurvnnfNnubtKxtVQve5iEnHs5B9bDAZbwC1prP, and QmRGUzBxZzHP4L4K44WMHSWMdU1RTMLqGrWypGrtCuz9R3

paymog commented 1 year ago

This was fixed in firehose version 1.3.2