Closed andygrove closed 9 months ago
Hi @andygrove, we also meet the same issue. I will propose a PR to add a config to make the maximum decoded message size configurable for temporary fix.
I am still running into this error with the latest code.
2023-12-11T14:31:18.347839Z WARN task_runner ThreadId(82) ballista_executor::cpu_bound_executor: Spawned task output ignored: receiver dropped
2023-12-11T14:31:18.484649Z WARN tokio-runtime-worker ThreadId(45) ballista_executor::execution_loop: Executor poll work loop failed. If this continues to happen the Scheduler might be marked as dead. Error: status: OutOfRange, message: "Error, message length too large: found 7700152 bytes, the limit is: 4194304 bytes", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Mon, 11 Dec 2023 14:31:18 GMT"} }
I am using the default --grpc-server-max-decoding-message-size
size of 16 MB, but the limit still appears to be 4 MB.
We currently set the decoding max size but not the encoding max size, so perhaps that is the issue. I will test this.
We've hit some other errors related to max sizes at our end (Coralogix), we reduced those errors by:
Some other things we did:
PruneUnusedPartitions
, as our plans can contain 1000s of partitions.I confirmed that setting the max encoding size resolves the issue for me.
We set max encode/decode message size when creating the gRPC servers, but not for the clients, so I ran into this again.
Describe the bug
I tried running some benchmarks, but some queries fail with this error:
To Reproduce
Start cluster:
Run TPC-H benchmarks
Expected behavior Should not fail
Additional context