Unable to execute query with Multi-Stage Engine

[vineethvp]

Intermittently getting below error while running certain queries. Segments seems to be healthy and cluster is also running fine. Mostly observed this with join queries ,so should be related to multi stage query engine. Nothing could be traced in the logs directing to a root cause. Please suggest what might be causing this error?

Error Code: 200
QueryExecutionError:
Unable to execute query plan for request: 1489691015000000078 on server: pinot-server-1.pinot-server-headless.pinot-quickstart.svc.cluster.local@{38061,44933}, ERROR: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Caught exception while deserializing stage plan for request: 1489691015000000078, stage: 1
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2028)
    at org.apache.pinot.query.service.server.QueryServer.submit(QueryServer.java:156)
    at org.apache.pinot.common.proto.PinotQueryWorkerGrpc$MethodHandlers.invoke(PinotQueryWorkerGrpc.java:284)
...
Caused by: java.lang.RuntimeException: Caught exception while deserializing stage plan for request: 1489691015000000078, stage: 1
    at org.apache.pinot.query.service.server.QueryServer.lambda$submit$1(QueryServer.java:126)
    at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
    ... 3 more
Caused by: java.lang.IllegalStateException: Unable to create Object of type: org.apache.calcite.rel.logical.PinotRelExchangeType
    at org.apache.pinot.query.planner.serde.ProtoSerializationUtils.constructObject(ProtoSerializationUtils.java:252)
    at org.apache.pinot.query.planner.serde.ProtoSerializationUtils.constructMemberVariable(ProtoSerializationUtils.java:194)
    at org.apache.pinot.query.planner.serde.ProtoSerializationUtils.setObjectFieldToObject(ProtoSerializationUtils.java:59)
    at org.apache.pinot.query.planner.plannode.AbstractPlanNode.fromObjectField(AbstractPlanNode.java:81)
...
Caused by: java.lang.ClassNotFoundException: org.apache.calcite.rel.logical.PinotRelExchangeType
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
    at java.base/java.lang.Class.forName0(Native Method)
org.apache.pinot.query.service.dispatch.QueryDispatcher.submit(QueryDispatcher.java:206)
org.apache.pinot.query.service.dispatch.QueryDispatcher.submitAndReduce(QueryDispatcher.java:97)
org.apache.pinot.broker.requesthandler.MultiStageBrokerRequestHandler.handleRequest(MultiStageBrokerRequestHandler.java:199)
org.apache.pinot.broker.requesthandler.BaseBrokerRequestHandler.handleRequest(BaseBrokerRequestHandler.java:290)

[vineethvp]

This issue pertains to one table alone. below select query with 'Use Multi-Stage Engine' also giving same error.

But it works for other tables.

select * from kpi_events_stg where upload_ts_millis > ago('PT180M')

@Jackie-Jiang Please provide if you have any sugestions.

[Jackie-Jiang]

This is very likely caused by the backward incompatibility introduced in #12837, which is then fixed in #12944. Can you try rebuild pinot with the latest master?

[vineethvp]

@Jackie-Jiang Can we rebuild without losing the data and other configurations? Could you please provide some reference on upgrading the cluster? cluster is currently running on k8 without deepstorage.

[Jackie-Jiang]

I don't follow. Rebuild a project shouldn't cause any data lost. You might find more help in the slack troubleshooting channel

[vineethvp]

@Jackie-Jiang Sure, Thanks.