cannot set mem_limit query option

[fafnirbcrow]

I am trying to connect to a cloudera instance of impala, which expects me to set mem_limit as a query option, otherwise it will refuse too large running of queries.

I was attempting to implement this last week with the beeswax version and had forked to be able to set beeswax configuration options, but I was having problems getting it to pass the right string.

With the switch to HS2, I thought by swapping to the ODBC, I could send it as a separate query before and it would keep it in session potentially for the user, but this is not the case.

I can see where the mem_limit option is part of the thrift definition https://github.com/bippio/go-impala/blob/fced9e97c0fae24a795b5d20477cc4a65c9098cd/services/impalaservice/ImpalaService.go#L36, but I cannot seem to sort out how to properly send it, and could not sort out how it used those or the default options in the code.

Can someone clarify how to properly make those settings?

[shaloi]

you can set it as a query as below:

ctx := context.Background()
rows, err = db.QueryContext(ctx, "set mem_limit=3m") // set mem_limit to 3MB
if err != nil {
   log.Error(err)
}

Impala query memory limit can also be set server side as a impalad startup option. Please have a look at this: https://www.cloudera.com/documentation/enterprise/5-16-x/topics/impala_mem_limit.html

[fafnirbcrow]

I had attempted that thinking it would set it with the query session that way, but it does not seem to work. I still receive the 'Rejected query from pool root.default: request memory needed 108.00 GB is greater than pool max mem resources 100.00 GB'

I unfortunately dont have access or ability to change the startup.

Is there a way to change the request pool maybe? They may have different settings per pool connection

[intamyuto]

I submitted PR to add 'MEM_LIMIT' option.

You can supply mem-limit option for the connection pool:

db, err := sql.Open("impala", "impala://serveraddr?mem-limit=3m")

[fafnirbcrow]

That solved the problem. You guys are awesome! Thank you so much for the help!