Seeing an issue with dynamic table queries through the Pinot connector for Trino version 379. When doing a query with a WHERE clause on the partition key without using a dynamic table query where there is no data for that key, 0 rows are returned (as expected). But when doing a dynamic table query of the same kind, the Pinot Trino connector hangs for a long time, and then returns
Query 20220920_202500_00441_6rt9p failed: Only 0 out of 0 servers responded for query select "field1", "field2", "field3", "field4", "field5", "time", "field6", "time2", "field7", "primaryKey", "field8", "field9" from table where ("primaryKey") = 'primary-key-which-does-not-exist' limit 10 with query .......
This is a result of executing the following trino query
SELECT * FROM pinot.default."SELECT anyField FROM table WHERE primaryKey = 'primary-key-which-does-not-exist'";
Executing the same query outside of a dynamic table query works as expected, and returns 0 rows. In the Trino UI the error shows up as EXTERNAL ERROR — PINOT_INSUFFICIENT_SERVER_RESPONSE
Here is a stack trace
at io.trino.plugin.pinot.client.PinotClient.lambda$submitBrokerQueryJson$6(PinotClient.java:551)
at io.trino.plugin.pinot.client.PinotClient.doWithRetries(PinotClient.java:616)
at io.trino.plugin.pinot.client.PinotClient.submitBrokerQueryJson(PinotClient.java:532)
at io.trino.plugin.pinot.client.PinotClient.createResultIterator(PinotClient.java:573)
at io.trino.plugin.pinot.PinotBrokerPageSource.getNextPage(PinotBrokerPageSource.java:112)
at io.trino.operator.ScanFilterAndProjectOperator$ConnectorPageSourceToPages.process(ScanFilterAndProjectOperator.java:392)
at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
at io.trino.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:240)
at io.trino.operator.WorkProcessorUtils$YieldingProcess.process(WorkProcessorUtils.java:182)
at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
at io.trino.operator.WorkProcessorUtils.lambda$flatten$7(WorkProcessorUtils.java:296)
at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:338)
at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
at io.trino.operator.WorkProcessorUtils$3.process(WorkProcessorUtils.java:325)
at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
at io.trino.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:240)
at io.trino.operator.WorkProcessorUtils.lambda$processStateMonitor$3(WorkProcessorUtils.java:219)
at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
at io.trino.operator.WorkProcessorUtils.getNextState(WorkProcessorUtils.java:240)
at io.trino.operator.WorkProcessorUtils.lambda$finishWhen$4(WorkProcessorUtils.java:234)
at io.trino.operator.WorkProcessorUtils$ProcessWorkProcessor.process(WorkProcessorUtils.java:391)
at io.trino.operator.WorkProcessorSourceOperatorAdapter.getOutput(WorkProcessorSourceOperatorAdapter.java:150)
at io.trino.operator.Driver.processInternal(Driver.java:388)
at io.trino.operator.Driver.lambda$processFor$9(Driver.java:292)
at io.trino.operator.Driver.tryWithLock(Driver.java:693)
at io.trino.operator.Driver.processFor(Driver.java:285)
at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1092)
at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:488)
at io.trino.$gen.Trino_379____20220920_201245_2.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
In the Pinot broker itself, I can see these messages
No server found for request 99: select.....
Interestingly enough, this seems to only happen if the entire partition is empty. So for example, if you have a Pinot table with 10 partitions, where partitions 0,1,2,3,4 have data, but partition 5,6,7,8,9 do not, then any query with a key which maps to the empty partitions will fail.
This means that the issue is not predicated on whether a table has data for the specified key or not, but whether the partition has any data at all.
Workaround
I have worked around this issue by reducing the amount of partitions for my tables to 1. After this change, any query with a key which does not exist just returns 0 rows, as it maps to a partition which does have data, just not for that key.
Seeing an issue with dynamic table queries through the Pinot connector for Trino version 379. When doing a query with a WHERE clause on the partition key without using a dynamic table query where there is no data for that key, 0 rows are returned (as expected). But when doing a dynamic table query of the same kind, the Pinot Trino connector hangs for a long time, and then returns
Query 20220920_202500_00441_6rt9p failed: Only 0 out of 0 servers responded for query select "field1", "field2", "field3", "field4", "field5", "time", "field6", "time2", "field7", "primaryKey", "field8", "field9" from table where ("primaryKey") = 'primary-key-which-does-not-exist' limit 10 with query .......
This is a result of executing the following trino query
SELECT * FROM pinot.default."SELECT anyField FROM table WHERE primaryKey = 'primary-key-which-does-not-exist'";
Executing the same query outside of a dynamic table query works as expected, and returns 0 rows. In the Trino UI the error shows up as
EXTERNAL ERROR — PINOT_INSUFFICIENT_SERVER_RESPONSE
Here is a stack trace
In the Pinot broker itself, I can see these messages
No server found for request 99: select.....
Interestingly enough, this seems to only happen if the entire partition is empty. So for example, if you have a Pinot table with 10 partitions, where partitions 0,1,2,3,4 have data, but partition 5,6,7,8,9 do not, then any query with a key which maps to the empty partitions will fail.
This means that the issue is not predicated on whether a table has data for the specified key or not, but whether the partition has any data at all.
Workaround I have worked around this issue by reducing the amount of partitions for my tables to 1. After this change, any query with a key which does not exist just returns 0 rows, as it maps to a partition which does have data, just not for that key.