Ran out of free wasm instances, when using `runtime_api`

[seunlanlege]

Is there an existing issue?

• [X] I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• [X] This is not a support question.

Description of bug

In the Hyperbridge node, we provide an RPC for querying ISMP events which calls into the runtime using the runtime api.

https://github.com/polytope-labs/hyperbridge/blob/main/modules/ismp/pallets/rpc/src/lib.rs#L423

The problem is if this RPC gets called with a very large block range or perhaps too frequently, users start to observe the following error:

Which eventually causes the node to fall out of sync.

Steps to reproduce

No response

[tmpolaczyk]

I believe this is harmless, as a new instance will be allocated anyway. See here, maybe someone more familiar with the code can confirm.

https://github.com/paritytech/polkadot-sdk/blob/aff3a0796176ff3c0ee1b89c2f1d811a858f17a8/substrate/client/executor/src/wasm_runtime.rs#L143

Not sure why the warning exists, can it be removed?

[seunlanlege]

I believe this is harmless, as a new instance will be allocated anyway. See here, maybe someone more familiar with the code can confirm.

My question is why does it stall block syncing?

[bkchr]

Does it really stall or is it just extremely slow? Tell them to set --max-runtime-instances 32 and see if that helps.

[bkchr]

https://github.com/paritytech/substrate/issues/8846 generally this would prevent this by having some priority based cache for consensus related actions.

[seunlanlege]

Does it really stall or is it just extremely slow?

It actually stalls, from the attached image. Its syncing at 0.0bps, while serving a lot of rpc requests

[bkchr]

Can you please ask them to run with -lsync=trace and post the logs when it stalls?