search

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.49k stars 3.02k forks source link

Trino worker may crash when using arbitrary aggregation function #15285

Open losipiuk opened 1 year ago

losipiuk commented 1 year ago

Using GROUP BY arbitrary(column) you may easily lead to Trino workers crash as the state memory usage is underestimated. Memory usage reported by the query is smaller than actual usage.

Repro. Run MemoryQueryRunner with -Xmx8g and execute:

set session query_max_memory='2GB';
select * from (select 2*orderkey, arbitrary(comment) from tpch.sf100.lineitem group by 2*orderkey) limit 10;

query will result in an OOM.

This is mostly visible with VARCHAR(X) columns Quick debugging session suggests that generated GroupedAccumulator.getEstimatedSize() returns 24. This totally declared length of VARCHAR type.

losipiuk commented 1 year ago

@dain could that be result of recent aggregation functions refactorings or it was always like that?

losipiuk commented 1 year ago

Note: it may not be an exactly same problem but it is easy to get OOM also with other aggregating queries with very high group by cardinality. Observed crashes with max on bigint data type

select * from (select 2*l_orderkey, max(3*l_orderkey), max(7*l_orderkey), max(l_orderkey) from lineitem group by 2*l_orderkey) limit 10