DRILL-8426: Fix endless retrying zk set data for a large query

Description

Zookeeper closes a connection with a client if he tries to set data with a size bigger than jute.maxbuffer. By default, it is equal to 1MB.

Drill persists it's running queries in zookeeper. If you issue a large query (bigger than the value of jute.maxbuffer on the zookeeper server) Drill will try to persist it and get a ConnectinLoss exception, curator (client library that Drill uses to communicate with Zookeeper) will try to retry the set command based on RetryPolicy. Drill uses RetryNTimes policy which in Drill is set so to keep retrying for 7200 times. And while Drill retrying to persist large query to zookeeper, he with each try will losing connection with zookeeper (server will cutting off connection, because data has to big size) and it will keeping for around 1 hour. After this, the client that issued the big query will not receive any error or any result, cause the final exception is not properly processed.

What I change:

Drill will compare the size of data with the client jute.maxbuffer value and if it is bigger, then throw IllegalArgumentException that will be wrapped into UserException.executionError. It is still doesn't safe Drill from trying to persist to big data into zookeeper, because a user can manually change the value of jute.maxbuffer on the client or the server side and then may have inconsistent values (a client jute.maxbuffer value is not equal to a server jute.maxbuffer). But as said in zookeeper documentation, if the user has changed jute.maxbuffer value, then the user should change it on all the zookeeper servers and clients. So in the general case - this check will be enough.
Make Foreman properly process exception that may be raised from queryStateProcessor.moveToState.
Reduce drill.exec.zk.retry.count from 7200 to 15
Add info logs, if the zookeeper client will raise exception during set operation, so the user was aware what the data size was and what value of jute.maxbuffer Drill has.

Documentation

Add some information to troubleshooting page, what to do if you catched such an exception and Drill was not responding for a long time?

Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /drill/running/1bb91a06-3afe-8152-f3ce-048dd3bef992
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1672)
        at org.apache.curator.framework.imps.CreateBuilderImpl$18.call(CreateBuilderImpl.java:1216)
        at org.apache.curator.framework.imps.CreateBuilderImpl$18.call(CreateBuilderImpl.java:1193)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:1190)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:605)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:595)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:48)
        at org.apache.drill.exec.coord.zk.ZookeeperClient.put(ZookeeperClient.java:294)
        ... 10 common frames omitted

Testing

Manual test, I tried to execute a huge query like this:

select full_name from cp.`employee.json` where full_name in ('Sheri Nowmer', 'Sheri Nowmer', ........)

apache / drill