search

RunningJon / TPC-DS

Greenplum TPC-DS benchmark
113 stars 96 forks source link

Insufficient memory reserved for statement #6

Closed aksh-4793 closed 7 years ago

aksh-4793 commented 7 years ago

When I run the script, the 4th query ( 104.tpcds.04.sql ) throws me the following error:

psql: TPC-DS/05_sql/104.tpcds.04.sql:115: ERROR: insufficient memory reserved for statement HINT: Increase statement memory or reduce the number of Parquet tables to be scanned.

RunningJon commented 7 years ago

How much memory do you have in each node of your cluster? How much ram is used for other processes? How much swap do you have? How much memory do you have allocated to hawq? What is the over commit value and ratio?

This error is caused by a memory configuration problem in hawq and not in the query.

Jon Roberts

On Jan 2, 2017, at 3:52 AM, aksh-4793 notifications@github.com wrote:

When I run the script, the 4th query ( 104.tpcds.04.sql ) throws me the following error:

psql: TPC-DS/05_sql/104.tpcds.04.sql:115: ERROR: insufficient memory reserved for statement HINT: Increase statement memory or reduce the number of Parquet tables to be scanned.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

aksh-4793 commented 7 years ago

I have setup 4 nodes 1 master and 3 segments on Ubuntu The slaves have about 128GB RAM

The parameters I have set in HAWQ GUC are as follows: hawq_re_memory_overcommit_max : 98304 hawq_rm_stmt_vseg_memory : 16gb

In my kernel parameters I have overcommit ratio as 80

Let me know if I need to specify anything else.

RunningJon commented 7 years ago

Change the hawq_rm_stmt_vseg_memory to the default of 128mb.

Do you have swap? You should have 4GB of swap.

hawq_rm_memory_limit_perseg sholud be set to 120 or a bit lower if you have a lot of other processes going on these nodes.

Overcommit ratio should be 100 because you have 128GB of RAM.

On Fri, Jan 6, 2017 at 3:49 AM, aksh-4793 notifications@github.com wrote:

I have setup 4 nodes 1 master and 3 segments on Ubuntu The slaves have about 128GB RAM

The parameters I have set in HAWQ GUC are as follows: hawq_re_memory_overcommit_max : 98304 hawq_rm_stmt_vseg_memory : 16gb

In my kernel parameters I have overcommit ratio as 80

Let me know if I need to specify anything else.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pivotalguru/TPC-DS/issues/6#issuecomment-270868050, or mute the thread https://github.com/notifications/unsubscribe-auth/AFObRHeJgIZmtUBkEccfN-ZvrufT3Ux7ks5rPg4mgaJpZM4LY5Hh .