maxRequestsQueuedPerDestination should be exposed and listed in Presto config.properties

[linusguan]

Hi,

I recently encountered an issue when using Presto. java.sql.SQLException: Query failed (#20170511_094135_03925_r94jh): Max requests per destination 1024 exceeded for HttpDestination[http://10.243.61.123:8889]@3709d005,queue=1024,pool=DuplexConnectionPool[c=250/250,a=0,i=0]

After some research I think it might be related to Jetty default connection limit, should I increse following values? I found them here, I think they are not documented.

exchange.http-client.max-connections=1000
exchange.http-client.max-connections-per-server=1000
scheduler.http-client.max-connections=1000
scheduler.http-client.max-connections-per-server=1000

Can anyone help me with this and explain what does these parameters do? Thanks!

[linusguan]

Recently found out these options in upstream repo airlift

Is there a presto config property to alter this: private int maxRequestsQueuedPerDestination = 1024; ?

[nod]

Any update on this? We are seeing the same issue on presto on emr.

[linusguan]

@nod We upped the size of our emr master instance size so a query can be processed faster thus the connection pool won't fill up. It's a workaround.

[nod]

@linusguan mind sharing what size you're on? I know it won't be apples to apples but it's an interesting datapoint.

edit - we're on 20 nodes @ m4.2xl btw

[linusguan]

@nod Our load is extremely high and we ended up using m4.10xlarge for master :P It's probably is a pretty unique use case.

[wgzhao]

I meet the same problem. My SQL statement consists of large number of "union all" sub queries. like the following: select col1 from tbl1 where clause union all select col1 from tb1 where clause .......... The number of "union all" in my case more than 730.

[deanet]

any update of this issue ?

[nezihyigitbasi]

Setting the below config parameters should configure the clients to handle more connections (where <client_name> is exchange/scheduler):

<client_name>.http-client.max-requests-queued-per-destination
<client_name>.http-client.max-connections-per-server

[deanet]

@nezihyigitbasi

presto can't run.

2018-03-13T03:22:17.328Z WARN main Bootstrap UNUSED PROPERTIES 2018-03-13T03:22:17.328Z WARN main Bootstrap http-client.max-connections-per-server 2018-03-13T03:22:17.329Z WARN main Bootstrap http-client.max-requests-queued-per-destination 2018-03-13T03:22:17.329Z WARN main Bootstrap
2018-03-13T03:22:18.413Z ERROR main com.facebook.presto.server.PrestoServer Unable to create injector, see the following errors:

1) Configuration property 'http-client.max-connections-per-server' was not used at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:235)

2) Configuration property 'http-client.max-requests-queued-per-destination' was not used at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:235)

2 errors com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Configuration property 'http-client.max-connections-per-server' was not used at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:235)

2) Configuration property 'http-client.max-requests-queued-per-destination' was not used at io.airlift.bootstrap.Bootstrap.lambda$initialize$2(Bootstrap.java:235)

2 errors at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:466) at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155) at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107) at com.google.inject.Guice.createInjector(Guice.java:96) at io.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:242) at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:116) at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:67)

i'm working on EMR with release 5.12.0

[deanet]

ok it's running. thank you

[dletendre]

what properties did you set? i assume you edited /etc/presto/conf/config.properities on the EMR master?

[deanet]

@dletendre yup, that's right.

for slave/worker:

sudo su -c 'echo "exchange.http-client.max-requests-queued-per-destination=5000
exchange.http-client.max-connections-per-server=5000" >> /etc/presto/conf/config.properties'
sudo restart presto-server

for master/node:

sudo su -c 'echo "scheduler.http-client.max-requests-queued-per-destination=5000
scheduler.http-client.max-connections-per-server=5000
exchange.http-client.max-requests-queued-per-destination=5000
exchange.http-client.max-connections-per-server=5000" >> /etc/presto/conf/config.properties'
sudo restart presto-server

[coalchan]

just add property in config.properties of coordinator node : scheduler.http-client.max-requests-queued-per-destination=2048 succeed at version 0.168 !

detail link: https://zhuanlan.zhihu.com/p/57956341

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 2 years. If you feel that this issue is important, just comment and the stale tag will be removed; otherwise it will be closed in 7 days. This is an attempt to ensure that our open issues remain valuable and relevant so that we can keep track of what needs to be done and prioritize the right things.