julianhyde / sqlline

Shell for issuing SQL to relational databases via JDBC
BSD 3-Clause "New" or "Revised" License
623 stars 147 forks source link

Unable to set hive parameter #238

Open mtdeguzis opened 5 years ago

mtdeguzis commented 5 years ago

Works via normal on-cluster with Hortonworks Hadoop beeline, so I assume I am missing something simple...

beeline:

Connecting to jdbc:hive2://host.domain.com:00000/;principal=hive/host.domain.com@DOMAIN.COM
Connected to: Apache Hive (version 1.2.1000.2.6.5.90-1)
Driver: Hive JDBC (version 1.2.1000.2.6.5.90-1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
No rows affected (0.044 seconds)

sqlline:

/sqlline -u "jdbc:hive2://host.domain.com:00000/;AuthMech=1;KrbHostFQDN=adcontroller.domain.com;KrbServiceName=hive;KrbHostFQDN=host.domain.com;KrbRealm=HOST.COM"

0: jdbc:hive2://host.domain.com:>  set hive.execution.engine=tez;
Error: [Simba][HiveJDBCDriver](500051) ERROR processing query/statement. Error Code: 1, SQL state: TStatus(statusCode:ERROR_STATUS, infoMessages:[*org.apache.hive.service.cli.HiveSQLException:Error while processing statement: Cannot modify execution.engine at runtime. It is not in list of params that are allowed to be modified at runtime:27:26, org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:324, org.apache.hive.service.cli.operation.HiveCommandOperation:runInternal:HiveCommandOperation.java:108, org.apache.hive.service.cli.operation.Operation:run:Operation.java:264, org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:479, org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementAsync:HiveSessionImpl.java:466, sun.reflect.GeneratedMethodAccessor66:invoke::-1, sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43, java.lang.reflect.Method:invoke:Method.java:498, org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:78, org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:36, org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:63, java.security.AccessController:doPrivileged:AccessController.java:-2, javax.security.auth.Subject:doAs:Subject.java:422, org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1869, org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:59, com.sun.proxy.$Proxy44:executeStatementAsync::-1, org.apache.hive.service.cli.CLIService:executeStatementAsync:CLIService.java:315, org.apache.hive.service.cli.thrift.ThriftCLIService:ExecuteStatement:ThriftCLIService.java:509, org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1377, org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement:getResult:TCLIService.java:1362, org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39, org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39, org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor:process:HadoopThriftAuthBridge.java:562, org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286, java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1149, java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:624, java.lang.Thread:run:Thread.java:748], sqlState:42000, errorCode:1, errorMessage:Error while processing statement: Cannot modify execution.engine at runtime. It is not in list of params that are allowed to be modified at runtime), Query: set execution.engine=tez. (state=HY000,code=500051)

The property is listed with a normal set command under sqlline:

| hive.execution.engine=tez                                                                                                                                                                            
| hive.mapred.reduce.tasks.speculative.execution=false                                                                                                                                                 
| hive.vectorized.execution.enabled=true                                                                                                                                                               
| hive.vectorized.execution.mapjoin.minmax.enabled=false                                                                                                                                               
| hive.vectorized.execution.mapjoin.native.enabled=true                                                                                                                                                
| hive.vectorized.execution.mapjoin.native.fast.hashtable.enabled=false                                                                                                                                
| hive.vectorized.execution.mapjoin.native.multikey.only.enabled=false                                                                                                                                
| hive.vectorized.execution.mapjoin.overflow.repeated.threshold=-1                                                                                                                                     
| hive.vectorized.execution.reduce.enabled=true                                                                                                                                                       
| hive.vectorized.execution.reduce.groupby.enabled=true         

Seems we can't source the whitelist from the hiveserver:

+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                                                                               set                                                                    |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| security.authorization.sqlstd.confwhitelist is undefined                                                                                                                                             |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row selected (0.087 seconds)
sqlline version 1.5.0-SNAPSHOT
Closing: com.simba.hiveserver2.jdbc.jdbc41.S41Connection

... When that normally has many properties listed.

djmartin6 commented 5 years ago

To add more info to this ticket. It seems that there is a workaround. When running set hive.execution.engine=mr; it fails but if we do set hive.hive.execution.engine=mr; it sets the correct value almost as if it's removing the hive. at the beginning of the parameter before running the query.

snuyanzin commented 5 years ago

I wonder why are you using sqlline version 1.5.0-SNAPSHOT I tried the similar case with sqlline version 1.6.0, HDP-2.6.5, jdbc driver hive-jdbc-1.2.1000.2.6.5.0-292-standalone.jar but with much simpler connection string like

sqlline> !connect jdbc:hive2://localhost:10000/default "" "" ""                                                                                                                                                    
Dec 27, 2018 10:31:44 AM org.apache.hive.jdbc.Utils parseURL                                                                                                                                                       
INFO: Supplied authorities: localhost:10000                                                                                                                                                                        
Dec 27, 2018 10:31:44 AM org.apache.hive.jdbc.Utils parseURL                                                                                                                                                       
INFO: Resolved authority: localhost:10000                                                                                                                                                                          
Transaction isolation level TRANSACTION_REPEATABLE_READ is not supported. Default (TRANSACTION_NONE) will be used instead.                                                                                         
0: jdbc:hive2://localhost:10000/default> !set version                                                                                                                                                              
version             sqlline version 1.6.0                                                                                                                                                                          
0: jdbc:hive2://localhost:10000/default>                                                                                                                                                                           

I tried mentioned commands for setting and viewing hive variables

set hive.execution.engine=mr;
set hive.execution.engine=tez;
set;

and all of them look ok (see output below)

Are you sure you need to use sqlline version 1.5.0-SNAPSHOT rather than stable? Are you sure you do not have the similar issue while working with beeline (I do not know but may be properties from your connection string could play some role here)

the output of mentioned commands in case of simple connection string (the output of set is cut)

0: jdbc:hive2://localhost:10000/default> set;                                                                                                                                                                      
+-----+                                                                                                                                                                                                            
| set |                                                                                                                                                                                                            
+-----+                                                                                                                                                                                                            
| _hive.hdfs.session.path=/tmp/hive/hive/7a831ab3-0a15-4afd-83c8-50a75d842e82 |                                                                                                                                    
| _hive.local.session.path=/tmp/hive/7a831ab3-0a15-4afd-83c8-50a75d842e82 |                                                                                                                                        
| _hive.tmp_table_space=/tmp/hive/hive/7a831ab3-0a15-4afd-83c8-50a75d842e82/_tmp_space.db |                                                                                                                        
| ambari.hive.db.schema.name=hive |                                                                                                                                                                                
| atlas.cluster.name=Sandbox |                                                                                                                                                                                     
| atlas.hook.hive.maxThreads=1 |                                                                                                                                                                                   
| atlas.hook.hive.minThreads=1 |                                                                                                                                                                                   
| atlas.rest.address=http://sandbox-hdp.hortonworks.com:21000 | 
...
| system:user.home=/home/hive |                                                                                                                                                                                    
| system:user.language=en |                                                                                                                                                                                        
| system:user.name=hive |                                                                                                                                                                                          
| system:user.timezone=UTC |                                                                                                                                                                                       
+-----+                                                                                                                                                                                                            
984 rows selected (0.201 seconds)                                                                                                                                                                                  
0: jdbc:hive2://localhost:10000/default> set hive.execution.engine=tez;                                                                                                                                            
No rows affected (0.002 seconds)                                                                                                                                                                                   
0: jdbc:hive2://localhost:10000/default> set hive.execution.engine=mr;                                                                                                                                             
No rows affected (0.003 seconds)                                                                                                                                                                                   
0: jdbc:hive2://localhost:10000/default>