search

cloudera-labs / hms-mirror

"hms-mirror" is a utility used to bridge the gap between two clusters and migrate hive metadata.
Apache License 2.0
13 stars 8 forks source link

SQL Execution Issue with Cleanup #26

Closed dstreev closed 1 year ago

dstreev commented 1 year ago

hms-mirror v.1.5.4.1-SNAPSHOT commandline parameters: -d,SQL,-db,test_cdp,-tf,test_syntax,-e,-ma,--transfer-ownership,--accept

from the hms-mirror.log:

2023-02-14 12:54:40,958 [pool-7-thread-2] INFO Cluster.runTableSql(472):RIGHT:SQL:Set table owner:ALTER TABLE test_syntax SET OWNER USER hive
2023-02-14 12:54:41,044 [pool-7-thread-2] INFO Cluster.runTableSql(472):RIGHT:SQL:MSCK Repair Table:MSCK REPAIR TABLE hms_mirror_shadow_test_syntax
2023-02-14 12:54:41,377 [pool-7-thread-2] INFO Cluster.runTableSql(472):RIGHT:SQL:Moving data to new Namespace:FROM hms_mirror_shadow_test_syntax INSERT OVERWRITE TABLE test_syntax PARTITION (`cda_year`) SELECT * DISTRIBUTE BY `cda_year`
2023-02-14 12:55:02,861 [pool-7-thread-2] INFO Cluster.runTableSql(472):RIGHT:SQL:Dropping Shadow Table:DROP TABLE IF EXISTS hms_mirror_shadow_test_syntax
2023-02-14 12:55:02,975 [pool-7-thread-2] INFO Cluster.runTableSql(472):LEFT:SQL:Post Migration Cleanup:-- To be run AFTER final RIGHT SQL statements.
2023-02-14 12:55:02,992 [pool-7-thread-2] ERROR Cluster.runTableSql(479):org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: ParseException line 1:46 cannot recognize input near '<EOF>' '<EOF>' '<EOF>'
2023-02-14 12:55:02,992 [pool-7-thread-2] INFO Transfer.call(236):Migration complete for test_cdp.test_syntax in 35497ms

from the LEFT (HDP3) HS2 log:

2023-02-14T12:55:02,980  INFO [6c371e65-1aa7-4129-8c69-9ed9bb2f4f70 HiveServer2-HttpHandler-Pool: Thread-651243] operation.OperationManager: Adding operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=374f9928-7469-4e60-bde6-cf5e90d8866b]
2023-02-14T12:55:02,981  INFO [6c371e65-1aa7-4129-8c69-9ed9bb2f4f70 HiveServer2-HttpHandler-Pool: Thread-651243] ql.Driver: Compiling command(queryId=hive_20230214125502_6df5fb76-1723-4ae4-8b53-16bc228b8651): -- To be run AFTER final RIGHT SQL statements.
2023-02-14T12:55:02,990 ERROR [6c371e65-1aa7-4129-8c69-9ed9bb2f4f70 HiveServer2-HttpHandler-Pool: Thread-651243] ql.Driver: FAILED: ParseException line 1:46 cannot recognize input near '<EOF>' '<EOF>' '<EOF>'
org.apache.hadoop.hive.ql.parse.ParseException: line 1:46 cannot recognize input near '<EOF>' '<EOF>' '<EOF>'
    at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:233)
    at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:79)
    at org.apache.hadoop.hive.ql.parse.ParseUtils.parse(ParseUtils.java:72)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:616)
    at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1858)
    at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1805)
    at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1800)
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
    at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
    at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:262)
    at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260)
    at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:565)
    at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:551)
    at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
    at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:566)
    at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
    at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    at org.apache.thrift.server.TServlet.doPost(TServlet.java:83)
    at org.apache.hive.service.cli.thrift.ThriftHttpServlet.doPost(ThriftHttpServlet.java:210)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:513)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.server.Server.handle(Server.java:539)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
    at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:259)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2023-02-14T12:55:02,990  INFO [6c371e65-1aa7-4129-8c69-9ed9bb2f4f70 HiveServer2-HttpHandler-Pool: Thread-651243] ql.Driver: Completed compiling command(queryId=hive_20230214125502_6df5fb76-1723-4ae4-8b53-16bc228b8651); Time taken: 0.009 seconds
2023-02-14T12:55:02,990  INFO [6c371e65-1aa7-4129-8c69-9ed9bb2f4f70 HiveServer2-HttpHandler-Pool: Thread-651243] lockmgr.DbLockManager: releaseLocks: []
2023-02-14T12:55:02,990  INFO [6c371e65-1aa7-4129-8c69-9ed9bb2f4f70 HiveServer2-HttpHandler-Pool: Thread-651243] operation.OperationManager: Closing operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=374f9928-7469-4e60-bde6-cf5e90d8866b]