Memory connector over-accounts memory allocations

[wenfengsui]

com.facebook.presto.spi.PrestoException: Memory limit [34359738368] for memory connector exceeded
    at com.facebook.presto.plugin.memory.MemoryPagesStore.add(MemoryPagesStore.java:68)
    at com.facebook.presto.plugin.memory.MemoryPageSinkProvider$MemoryPageSink.appendPage(MemoryPageSinkProvider.java:87)
    at com.facebook.presto.operator.TableWriterOperator.addInput(TableWriterOperator.java:203)
    at com.facebook.presto.operator.Driver.processInternal(Driver.java:309)
    at com.facebook.presto.operator.Driver.lambda$processFor$6(Driver.java:234)
    at com.facebook.presto.operator.Driver.tryWithLock(Driver.java:537)
    at com.facebook.presto.operator.Driver.processFor(Driver.java:229)
    at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:622)
    at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:624)
    at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:776)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

presto-server version: 0.170

presto-cli

create table memory.default.ds_pub_site_skc_info_d as select * from hive.default.ds_pub_site_skc_info_d where site_tp='shein';

Query 20190103_035125_00002_dyd27, FAILED, 21 nodes
Splits: 566 total, 30 done (5.30%)
0:07 [19.4M rows, 2.97GB] [2.69M rows/s, 423MB/s]

Query 20190103_035125_00002_dyd27 failed: Memory limit [34359738368] for memory connector exceeded

Query copied 2.97GB data, but memory connector reported exceeding limit of 34,359,738,368.

[findepi]

This topic has been already reported (under hard to find name) as https://github.com/prestodb/presto/issues/10337 and discussed there.