search

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.22k stars 439 forks source link

[GLUTEN-7760] Fix udf implicit cast & update doc #7852

Closed marin-ma closed 3 weeks ago

marin-ma commented 3 weeks ago
  1. Fix udf implicit type conversion not working, refine the UT.
  2. Update the hive udf example in UDF document
  3. Moved some general utility functions from WholeStageTransformerSuite into GlutenQueryTest
github-actions[bot] commented 3 weeks ago

https://github.com/apache/incubator-gluten/issues/7760

github-actions[bot] commented 3 weeks ago

Run Gluten Clickhouse CI on x86

github-actions[bot] commented 3 weeks ago

Run Gluten Clickhouse CI on x86

github-actions[bot] commented 3 weeks ago

Run Gluten Clickhouse CI on x86

marin-ma commented 3 weeks ago

@loneylee The CH UT failure seems very odd, and also cannot get passed after the re-run. I couldn't figure out the reason. Could you help to check it?

Here's the failed UT and its log

[2024-11-08T05:23:02.405Z] - GLUTEN-7542: Fix cache refresh *** FAILED ***
[2024-11-08T05:23:02.413Z]   org.apache.hadoop.ipc.RemoteException: No lease on /tpch-data/issue_7542/00000_0 (inode 17131): File does not exist. Holder DFSClient_NONMAPREDUCE_-1409055608_1 does not have any open files.
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3428)
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3518)
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3485)
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:786)
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:536)
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
[2024-11-08T05:23:02.413Z]  at java.security.AccessController.doPrivileged(Native Method)
[2024-11-08T05:23:02.413Z]  at javax.security.auth.Subject.doAs(Subject.java:421)
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
[2024-11-08T05:23:02.413Z]  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
[2024-11-08T05:23:02.413Z]   at org.apache.hadoop.ipc.Client.call(Client.java:1476)
[2024-11-08T05:23:02.413Z]   at org.apache.hadoop.ipc.Client.call(Client.java:1413)
[2024-11-08T05:23:02.413Z]   at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
[2024-11-08T05:23:02.413Z]   at com.sun.proxy.$Proxy42.complete(Unknown Source)
[2024-11-08T05:23:02.414Z]   at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:462)
[2024-11-08T05:23:02.414Z]   at sun.reflect.GeneratedMethodAccessor336.invoke(Unknown Source)
[2024-11-08T05:23:02.414Z]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2024-11-08T05:23:02.414Z]   at java.lang.reflect.Method.invoke(Method.java:498)
[2024-11-08T05:23:02.414Z]   at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
[2024-11-08T05:23:02.414Z]   at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
[2024-11-08T05:23:02.414Z]   ...
loneylee commented 3 weeks ago

Run Gluten Clickhouse CI on x86

loneylee commented 3 weeks ago

@loneylee The CH UT failure seems very odd, and also cannot get passed after the re-run. I couldn't figure out the reason. Could you help to check it?

Here's the failed UT and its log

[2024-11-08T05:23:02.405Z] - GLUTEN-7542: Fix cache refresh *** FAILED ***
[2024-11-08T05:23:02.413Z]   org.apache.hadoop.ipc.RemoteException: No lease on /tpch-data/issue_7542/00000_0 (inode 17131): File does not exist. Holder DFSClient_NONMAPREDUCE_-1409055608_1 does not have any open files.
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3428)
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3518)
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3485)
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:786)
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:536)
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
[2024-11-08T05:23:02.413Z]    at java.security.AccessController.doPrivileged(Native Method)
[2024-11-08T05:23:02.413Z]    at javax.security.auth.Subject.doAs(Subject.java:421)
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
[2024-11-08T05:23:02.413Z]    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
[2024-11-08T05:23:02.413Z]   at org.apache.hadoop.ipc.Client.call(Client.java:1476)
[2024-11-08T05:23:02.413Z]   at org.apache.hadoop.ipc.Client.call(Client.java:1413)
[2024-11-08T05:23:02.413Z]   at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
[2024-11-08T05:23:02.413Z]   at com.sun.proxy.$Proxy42.complete(Unknown Source)
[2024-11-08T05:23:02.414Z]   at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:462)
[2024-11-08T05:23:02.414Z]   at sun.reflect.GeneratedMethodAccessor336.invoke(Unknown Source)
[2024-11-08T05:23:02.414Z]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2024-11-08T05:23:02.414Z]   at java.lang.reflect.Method.invoke(Method.java:498)
[2024-11-08T05:23:02.414Z]   at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
[2024-11-08T05:23:02.414Z]   at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
[2024-11-08T05:23:02.414Z]   ...

Retry and passed.