Investigate slow substreams subgraph syncing

[mangas]

https://thegraph.com/hosted-service/subgraph/harsh9200/pancake-v2-swap-only was raised by Messari today as syncing slowly (30% in one month). It's just putting swaps in the database - we have made a suggestion that they remove the interface (which creates a uniqueness check), and also make the entity immutable, but even with that change it is still taking a while given the simplicity of the subgraph.

Deployment details

[mangas]

When trying to replicate this issue I came across a problem with the substream. It fails with the following:

Error: rpc error: code = Internal desc = error during init_stores_and_backprocess: run_parallel_process failed: parallel processing run: scheduler run: receiving stream resp: rpc error: code = Unknown desc = internal: step new irr: handler step new: execute modules: applying executor results "store_input_token_balances": execute: store wasm call: block 30086000: module "store_input_token_balances": general wasm execution failed: call: store "store_input_token_balances" became too big at 1073962100, maximum size: 1073741824 (recovered by wazero)
wasm stack trace:

It seems the substreams is using too much store space which is a fatal error on the substreams side.

The subgraph doesn't seem to be marked as failed based on the error received from the grpc endpoint

[mangas]

The GRPC endpoint does not provide a mechanism to inspect the determinism level for the error, this needs to be added on the endpoint.

message Error {
  string module = 1;
  string reason = 2;
  repeated string logs = 3;
  // FailureLogsTruncated is a flag that tells you if you received all the logs or if they
  // were truncated because you logged too much (fixed limit currently is set to 128 KiB).
  bool logs_truncated = 4;
}

https://github.com/streamingfast/substreams/issues/361 will address this issue on the substreams side.

[sduchesneau]

Note: that Error message from protobuf is currently NOT used by substreams. Only the grpc-level error is returned, with a StatusCode, from this list:

https://grpc.github.io/grpc/core/md_doc_statuscodes.html

The code "Invalid_Argument" (3) is currently the way that deterministic errors are reported back to the client.

[github-actions[bot]]

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.