search

prestodb / presto

The official home of the Presto distributed SQL query engine for big data
http://prestodb.io
Apache License 2.0
15.97k stars 5.35k forks source link

presto memory usage mechanism problem, after the execution of the calculation task, the memory usage is not reduced #22983

Open mengbaba3316 opened 3 months ago

mengbaba3316 commented 3 months ago

Hi, I am new to presto and I am having memory-related problems with presto and would like to ask for help. I configured presto with 8GB of jvm memory, every time I performed a query calculation, presto's memory usage gradually increased until I reached 8GB, but when some query calculations were finished, presto's memory usage did not decrease, I do not know why, here are my several guesses

  1. presto increases memory to the maximum limit of the jvm based on the number of tasks, then cleans up memory when it is full and new query computing tasks come in
  2. presto increases memory to the maximum limit of the jvm based on the number of tasks, then cleans up memory when it is full and new query computing tasks come in I hope you can help me, thank you very much
ZacBlanco commented 3 months ago

There are a number of reasons why this can occur. One of them being that Presto internally has many caches; It is possible some the caches or being filled during execution which increases the memory utilization. It should not be an issue though as they are flushed when memory is required. Additionally, it could also just be JVM garbage collection not kicking it to reduce the heap footprint.

Are you seeing memory-related errors when executing later queries? More specifics would help.