search

apache / kyuubi

Apache Kyuubi is a distributed and multi-tenant gateway to provide serverless SQL on data warehouses and lakehouses.
https://kyuubi.apache.org/
Apache License 2.0
2.11k stars 915 forks source link

[Bug] [KSHC] write remote failed with FileAlreadyExistsException #6781

Closed lordk911 closed 3 weeks ago

lordk911 commented 4 weeks ago

Code of Conduct

Search before asking

Describe the bug

When I try to write data to a remote hive partition table got FileAlreadyExistsException:

Spark version: 3.3.3

KSHC : 

ll /data/soft/spark3/jars/kyuubi*
lrwxrwxrwx 1 bigtop bigtop 79 Sep  5  2023 /data/soft/spark3/jars/kyuubi-extension-spark-3-3_2.12-1.7.1.jar -> /data/soft/extentions4spark3.3/kyuubi/kyuubi-extension-spark-3-3_2.12-1.7.1.jar
lrwxrwxrwx 1 bigtop bigtop 81 Sep  5  2023 /data/soft/spark3/jars/kyuubi-spark-authz_2.12-1.8.0-SNAPSHOT.jar -> /data/soft/extentions4spark3.3/authZ33/kyuubi-spark-authz_2.12-1.8.0-SNAPSHOT.jar
lrwxrwxrwx 1 bigtop bigtop 90 May 21 16:56 /data/soft/spark3/jars/kyuubi-spark-connector-hive_2.12-1.10.0-SNAPSHOT.jar -> /data/soft/extentions4spark3.3/kyuubi/kyuubi-spark-connector-hive_2.12-1.10.0-SNAPSHOT.jar
lrwxrwxrwx 1 bigtop bigtop 74 Apr 23  2024 /data/soft/spark3/jars/kyuubi-spark-lineage_2.12-1.8.1.jar -> /data/soft/extentions4spark3.3/lineage/kyuubi-spark-lineage_2.12-1.8.1.jar
lrwxrwxrwx 1 bigtop bigtop 69 Sep  6  2023 /data/soft/spark3/jars/kyuubi-util-1.8.0-SNAPSHOT.jar -> /data/soft/extentions4spark3.3/authZ33/kyuubi-util-1.8.0-SNAPSHOT.jar
lrwxrwxrwx 1 bigtop bigtop 80 Sep  6  2023 /data/soft/spark3/jars/kyuubi-util-scala_2.12-1.8.0-SNAPSHOT.jar -> /data/soft/extentions4spark3.3/authZ33/kyuubi-util-scala_2.12-1.8.0-SNAPSHOT.jar

sql:

CREATE TABLE test.kshc_target (c1 STRING, c2 STRING, c3 STRING) USING orc PARTITIONED BY (pcode STRING, pdate int);

insert overwrite dev.test.kshc_target partition(pcode,pdate) 
select logid,app_name,app_version,pcode,pdate from test.test_userlog_gux_func_add_hx2
where pcode='13073' and pdate>20221001 and pdate<20221010

spark executor error:

scala> spark.sql("insert overwrite dev.test.kshc_target partition(pcode,pdate) select logid,app_name,app_version,pcode,pdate from test.test_userlog_gux_func_add_hx2 where pcode='13073' and pdate>20221001 and pdate<20221010")
24/10/24 18:02:29 WARN HiveConf: HiveConf of name hive.metastore.uri.selection does not exist
24/10/24 18:02:29 WARN HiveConf: HiveConf of name hive.tez.cartesian-product.enabled does not exist
24/10/24 18:02:29 ERROR DatasetExtractor: class org.apache.spark.sql.catalyst.plans.logical.OverwriteByExpression is not supported yet. Please contact datahub team for further support. 
24/10/24 18:02:42 WARN TaskSetManager: Lost task 0.0 in stage 5.0 (TID 12) (bj2q10135.x.com executor 7): org.apache.hadoop.fs.FileAlreadyExistsException: /warehouse/tablespace/managed/hive/test.db/kshc_target/.hive-staging_hive_2024-10-24_18-02-29_514_4599524259026821678-1/-ext-10000/_temporary/0/_temporary/attempt_202410241802293861134321098295840_0000_m_000000_0/pcode=13073/pdate=20221008/part-00000-fbb73878-7779-41eb-9280-8aa0d8493a5c.c000 for client 10.0.101.35 already exists
        at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:380)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2453)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2351)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:774)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:286)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1271)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1250)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1232)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1170)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:556)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:553)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:567)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:494)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
        at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:94)
        at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:226)
        at org.apache.hadoop.hive.ql.io.orc.WriterImpl.<init>(WriterImpl.java:94)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createWriter(OrcFile.java:320)
        at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:103)
        at org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:173)
        at org.apache.spark.sql.execution.datasources.BaseDynamicPartitionDataWriter.writeRecord(FileFormatDataWriter.scala:328)
        at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:370)
        at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:339)
        at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442)
        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
        at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480)
        at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): /warehouse/tablespace/managed/hive/test.db/kshc_target/.hive-staging_hive_2024-10-24_18-02-29_514_4599524259026821678-1/-ext-10000/_temporary/0/_temporary/attempt_202410241802293861134321098295840_0000_m_000000_0/pcode=13073/pdate=20221008/part-00000-fbb73878-7779-41eb-9280-8aa0d8493a5c.c000 for client 10.0.101.35 already exists
        at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:380)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2453)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2351)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:774)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1612)
        at org.apache.hadoop.ipc.Client.call(Client.java:1558)
        at org.apache.hadoop.ipc.Client.call(Client.java:1455)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:129)
        at com.sun.proxy.$Proxy34.create(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:382)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at com.sun.proxy.$Proxy35.create(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:280)
        ... 32 more

24/10/24 18:02:46 WARN TaskSetManager: Lost task 1.0 in stage 5.0 (TID 13) (bj2q10135.x.com executor 7): org.apache.hadoop.fs.FileAlreadyExistsException: /warehouse/tablespace/managed/hive/test.db/kshc_target/.hive-staging_hive_2024-10-24_18-02-29_514_4599524259026821678-1/-ext-10000/_temporary/0/_temporary/attempt_202410241802293861134321098295840_0000_m_000001_0/pcode=13073/pdate=20221003/part-00001-fbb73878-7779-41eb-9280-8aa0d8493a5c.c000 for client 10.0.101.35 already exists
        at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:380)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2453)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2351)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:774)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:286)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1271)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1250)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1232)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1170)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:556)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:553)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:567)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:494)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
        at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:94)
        at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:226)
        at org.apache.hadoop.hive.ql.io.orc.WriterImpl.<init>(WriterImpl.java:94)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createWriter(OrcFile.java:320)
        at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:103)
        at org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:173)
        at org.apache.spark.sql.execution.datasources.BaseDynamicPartitionDataWriter.writeRecord(FileFormatDataWriter.scala:328)
        at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:370)
        at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:339)
        at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442)
        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
        at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480)
        at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): /warehouse/tablespace/managed/hive/test.db/kshc_target/.hive-staging_hive_2024-10-24_18-02-29_514_4599524259026821678-1/-ext-10000/_temporary/0/_temporary/attempt_202410241802293861134321098295840_0000_m_000001_0/pcode=13073/pdate=20221003/part-00001-fbb73878-7779-41eb-9280-8aa0d8493a5c.c000 for client 10.0.101.35 already exists
        at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:380)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2453)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2351)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:774)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1612)
        at org.apache.hadoop.ipc.Client.call(Client.java:1558)
        at org.apache.hadoop.ipc.Client.call(Client.java:1455)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:129)
        at com.sun.proxy.$Proxy34.create(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:382)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at com.sun.proxy.$Proxy35.create(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:280)
        ... 32 more

24/10/24 18:02:52 ERROR TaskSetManager: Task 0 in stage 5.0 failed 4 times; aborting job
24/10/24 18:02:52 ERROR OverwriteByExpressionExec: Data source write support org.apache.kyuubi.spark.connector.hive.write.HiveBatchWrite@61611b1f is aborting.
24/10/24 18:02:52 ERROR OverwriteByExpressionExec: Data source write support org.apache.kyuubi.spark.connector.hive.write.HiveBatchWrite@61611b1f aborted.
24/10/24 18:02:52 WARN TaskSetManager: Lost task 1.3 in stage 5.0 (TID 19) (bj2q10135.x.com executor 7): TaskKilled (Stage cancelled)
org.apache.spark.SparkException: Writing job aborted
  at org.apache.spark.sql.errors.QueryExecutionErrors$.writingJobAbortedError(QueryExecutionErrors.scala:767)
  at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:409)
  at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2$(WriteToDataSourceV2Exec.scala:353)
  at org.apache.spark.sql.execution.datasources.v2.OverwriteByExpressionExec.writeWithV2(WriteToDataSourceV2Exec.scala:262)
  at org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec.run(WriteToDataSourceV2Exec.scala:332)
  at org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec.run$(WriteToDataSourceV2Exec.scala:331)
  at org.apache.spark.sql.execution.datasources.v2.OverwriteByExpressionExec.run(WriteToDataSourceV2Exec.scala:262)
  at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
  at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
  at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
  at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
  at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
  at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
  at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
  at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
  at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
  at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
  at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
  at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
  at org.apache.spark.sql.Dataset.<init>(Dataset.scala:219)
  at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
  ... 47 elided
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in stage 5.0 (TID 18) (bj2q10135.x.com executor 7): org.apache.hadoop.fs.FileAlreadyExistsException: /warehouse/tablespace/managed/hive/test.db/kshc_target/.hive-staging_hive_2024-10-24_18-02-29_514_4599524259026821678-1/-ext-10000/_temporary/0/_temporary/attempt_202410241802293861134321098295840_0000_m_000000_0/pcode=13073/pdate=20221008/part-00000-fbb73878-7779-41eb-9280-8aa0d8493a5c.c000 for client 10.0.101.35 already exists
        at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:380)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2453)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2351)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:774)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:286)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1271)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1250)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1232)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1170)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:556)
        at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:553)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:567)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:494)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
        at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:94)
        at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:226)
        at org.apache.hadoop.hive.ql.io.orc.WriterImpl.<init>(WriterImpl.java:94)
        at org.apache.hadoop.hive.ql.io.orc.OrcFile.createWriter(OrcFile.java:320)
        at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:103)
        at org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:173)
        at org.apache.spark.sql.execution.datasources.BaseDynamicPartitionDataWriter.writeRecord(FileFormatDataWriter.scala:328)
        at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:370)
        at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:339)
        at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442)
        at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
        at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480)
        at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
        at org.apache.spark.scheduler.Task.run(Task.scala:136)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException): /warehouse/tablespace/managed/hive/test.db/kshc_target/.hive-staging_hive_2024-10-24_18-02-29_514_4599524259026821678-1/-ext-10000/_temporary/0/_temporary/attempt_202410241802293861134321098295840_0000_m_000000_0/pcode=13073/pdate=20221008/part-00000-fbb73878-7779-41eb-9280-8aa0d8493a5c.c000 for client 10.0.101.35 already exists
        at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:380)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2453)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2351)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:774)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1612)
        at org.apache.hadoop.ipc.Client.call(Client.java:1558)
        at org.apache.hadoop.ipc.Client.call(Client.java:1455)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242)
        at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:129)
        at com.sun.proxy.$Proxy34.create(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:382)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
        at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
        at com.sun.proxy.$Proxy35.create(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:280)
        ... 32 more

Driver stacktrace:
  at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2668)
  at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2604)
  at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2603)
  at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
  at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
  at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2603)
  at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1178)
  at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1178)
  at scala.Option.foreach(Option.scala:407)
  at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1178)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2856)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2798)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2787)
  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
  at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:952)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2238)
  at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:377)
  ... 82 more
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: /warehouse/tablespace/managed/hive/test.db/kshc_target/.hive-staging_hive_2024-10-24_18-02-29_514_4599524259026821678-1/-ext-10000/_temporary/0/_temporary/attempt_202410241802293861134321098295840_0000_m_000000_0/pcode=13073/pdate=20221008/part-00000-fbb73878-7779-41eb-9280-8aa0d8493a5c.c000 for client 10.0.101.35 already exists
        at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:380)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2453)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2351)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:774)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
  at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
  at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
  at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:286)
  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1271)
  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1250)
  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1232)
  at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1170)
  at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:556)
  at org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:553)
  at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:567)
  at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:494)
  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)
  at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
  at org.apache.orc.impl.PhysicalFsWriter.<init>(PhysicalFsWriter.java:94)
  at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:226)
  at org.apache.hadoop.hive.ql.io.orc.WriterImpl.<init>(WriterImpl.java:94)
  at org.apache.hadoop.hive.ql.io.orc.OrcFile.createWriter(OrcFile.java:320)
  at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat$OrcRecordWriter.write(OrcOutputFormat.java:103)
  at org.apache.spark.sql.hive.execution.HiveOutputWriter.write(HiveFileFormat.scala:173)
  at org.apache.spark.sql.execution.datasources.BaseDynamicPartitionDataWriter.writeRecord(FileFormatDataWriter.scala:328)
  at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:370)
  at org.apache.spark.sql.execution.datasources.DynamicPartitionDataSingleWriter.write(FileFormatDataWriter.scala:339)
  at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442)
  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
  at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480)
  at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:136)
  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hadoop.ipc.RemoteException: /warehouse/tablespace/managed/hive/test.db/kshc_target/.hive-staging_hive_2024-10-24_18-02-29_514_4599524259026821678-1/-ext-10000/_temporary/0/_temporary/attempt_202410241802293861134321098295840_0000_m_000000_0/pcode=13073/pdate=20221008/part-00000-fbb73878-7779-41eb-9280-8aa0d8493a5c.c000 for client 10.0.101.35 already exists
        at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:380)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2453)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2351)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:774)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

  at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1612)
  at org.apache.hadoop.ipc.Client.call(Client.java:1558)
  at org.apache.hadoop.ipc.Client.call(Client.java:1455)
  at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242)
  at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:129)
  at com.sun.proxy.$Proxy34.create(Unknown Source)
  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:382)
  at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
  at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
  at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
  at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
  at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
  at com.sun.proxy.$Proxy35.create(Unknown Source)
  at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:280)
  ... 32 more

hdfs file status:

hdfs dfs -count /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/*
           1            1              36284 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221001
           1            1              82203 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221002
           1            2             142246 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221003
           1            1             185693 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221004
           1            1             167230 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221005
           1            1             269932 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221006
           1            2             611056 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221007
           1            2             900962 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221008
           1            2             938944 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221009
           1            1             690660 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221010
           1            1            1191890 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221011
           1            1             806568 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221012
           1            3             674378 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221013
           1            2             830489 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221014
           1            1             275608 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221015
           1            1             280818 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221016
           1            1             840753 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221017
           1            2            1255153 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221018
           1            1            1461608 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221019
           1            2             920130 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221020
           1            2            1140300 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221021
           1            1             352938 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221022
           1            1             284561 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221023
           1            2             883231 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221024
           1            1            1732344 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221025
           1            1             923695 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221026
           1            1            1190956 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221027
           1            1             940066 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221028
           1            1             359856 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221029
           1            1             292837 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221030
           1            2            1572416 /warehouse/tablespace/managed/hive/test.db/test_userlog_gux_func_add_hx2/pcode=13073/pdate=20221031
scala> spark.sql("select count(1) from test.test_userlog_gux_func_add_hx2").show
24/10/24 18:13:21 ERROR DatasetExtractor: class org.apache.spark.sql.catalyst.plans.logical.Aggregate is not supported yet. Please contact datahub team for further support. 
+--------+                                                                      
|count(1)|
+--------+
|   91603|
+--------+

but when I choose on day data to write is ok: 

scala> spark.sql("insert overwrite dev.test.kshc_target partition(pcode,pdate) select logid,app_name,app_version,pcode,pdate from test.test_userlog_gux_func_add_hx2 where pcode='13073' and pdate=20221003")
24/10/24 18:07:01 ERROR DatasetExtractor: class org.apache.spark.sql.catalyst.plans.logical.OverwriteByExpression is not supported yet. Please contact datahub team for further support. 
res8: org.apache.spark.sql.DataFrame = []

Affects Version(s)

1.7.1

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

lordk911 commented 4 weeks ago 
scala> spark.sql("insert overwrite dev.test.kshc_target partition(pcode,pdate) select logid,app_name,app_version,pcode,pdate from test.test_userlog_gux_func_add_hx2 where pcode='13073' and pdate>=20221025 and pdate<=20221030")
24/10/25 12:54:30 ERROR DatasetExtractor: class org.apache.spark.sql.catalyst.plans.logical.OverwriteByExpression is not supported yet. Please contact datahub team for further support. 
res9: org.apache.spark.sql.DataFrame = []

The error is related to the number of partitions and the total number of files. If I continue to expand the time partition interval will report an error.

lordk911 commented 3 weeks ago

If I can keep that there is only one file in a partition, it will work

lordk911 commented 3 weeks ago

I've test against spark3.4.4, it can work out. It may be a bug about spark3.3.3, close issue.

pan3793 commented 3 weeks ago

this is a spark side issue, see details at SPARK-48484

lordk911 commented 3 weeks ago

I'm sorry to say that : 1、If I can keep that there is only one file in a partition, it will work (both spark3.3.3、spark3.4.4 on yarn or local) 2、if spark runs on local, no matter how many files in a partition, it will work, but will faild on yarn (both spark3.3.3、spark3.4.4)

bin/spark-shell --deploy-mode client --master local[*] : write remote is ok

bin/spark-shell --deploy-mode client --master yarn : only works when each partition only have one file , or will be failed with org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.fs.FileAlreadyExistsException)

pan3793 commented 3 weeks ago

if spark runs on local, no matter how many files in a partition, it will work, but will faild on yarn (both spark3.3.3、spark3.4.4)

failed with a "file already exists" error? or something else?

lordk911 commented 3 weeks ago

file already exists

yes, file already exists error.
also spark 3.5.3

lordk911 commented 3 weeks ago

may be it cause by FileWriterFactory use DynamicPartitionDataSingleWriter. It need : records to be written are required to be sorted on partition and/or bucket column(s) before writing.

/**
 * Dynamic partition writer with single writer, meaning only one writer is opened at any time for
 * writing. The records to be written are required to be sorted on partition and/or bucket
 * column(s) before writing.
 */
class DynamicPartitionDataSingleWriter(
    description: WriteJobDescription,
    taskAttemptContext: TaskAttemptContext,
    committer: FileCommitProtocol,
    customMetrics: Map[String, SQLMetric] = Map.empty)